you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 10 months ago* (3 children)

Oooooh. That's smart. I mostly host apps, but in theory, I should be able to dynamically modify the response body and tack on some HTML for a hidden button and do that.

I used to disallow everything in robots.txt but the worst crawlers just ignored it. Now my robots.txt says all are welcome and every bot gets shunted to the tarpit 😈

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 10 months ago (2 children)

    Nice! Thats another way to do it. 😀

    I know others use Arabis(?) I think thats what it called. The anime girl one that does a calc on top. Ive never had good luck with it. I think bot are using something to get around and it messes with my requests. Might also be my own fiddling.

  • source
  • parent
  • hideshow 2 child comments