Banning crawlers doesn't have to affect UX for humans. And you can whitelist your favorite crawlers too if that's your thing.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Banning crawlers doesn't have to affect UX for humans. And you can whitelist your favorite crawlers too if that's your thing.