I think AI can provide an interface to let user submit the site for crawling, such as some website scanner doing, like urlscan. Otherwise the site can reject the AI crawler.