you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 5 months ago

Why only news sites?

Any bot that ignores robots.txt should be banned.

  • source