53
Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs (blog.archive.org)
submitted 3 months ago by to c/opensource@lemmy.ml
8 comments fedilink hide all child comments
 

I guess Anubis also prevents "good" bots such as search engine bots to access a webpage...

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 months ago (2 children)

look in the crawlers directory and you can add any allows you want with the proper user agent. :)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago* (1 child)

    Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big "js-only" sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •