It's less hidden for admins. Whenever we click on a user's profile, there's a button that can take us directly to their individual mod-log history, which can help us quickly see if there's a history of AI or bot spam behavior that was spotted by other admins/mods.

Also did you mean “bot scrapers”

Agh, auto correct got me. Yes, I meant bot scrapers.

As for the robots.txt v2 idea, I'm not sure that would be very effective or popular amongst admins, as I think most would prefer not to give it any data if possible, and even with reduced rates, there would probably still be enough bots and queries to present a significant issue.

I think there have been some experiments with 'trapping' bots in a recursive loop that they don't realize is a loop, and therefore can't escape, but I'm not sure how effective those have been.