▲ 53 ▼ Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs (blog.archive.org) submitted 3 months ago by farooqkz@realbitcoin.cash to c/opensource@lemmy.ml 8 comments fedilink hide all child comments I guess Anubis also prevents "good" bots such as search engine bots to access a webpage...
[–] MonkderVierte@lemmy.zip 1 point 3 months ago (3 children) Can they please add "Dillo/3.2.0"? permalink fedilink source parent hideshow 3 child comments replies: [–] cupcakezealot@piefed.blahaj.zone 1 point 3 months ago (2 children) look in the crawlers directory and you can add any allows you want with the proper user agent. :) permalink fedilink source parent hideshow 2 child comments replies: [–] MonkderVierte@lemmy.zip 1 point 3 months ago* (1 child) Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big "js-only" sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible. permalink fedilink source parent hideshow 1 child comment replies: [–] cupcakezealot@piefed.blahaj.zone 2 points 3 months ago ooh i'm sorry i didn't catch that <3 permalink fedilink source parent
[–] cupcakezealot@piefed.blahaj.zone 1 point 3 months ago (2 children) look in the crawlers directory and you can add any allows you want with the proper user agent. :) permalink fedilink source parent hideshow 2 child comments replies: [–] MonkderVierte@lemmy.zip 1 point 3 months ago* (1 child) Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big "js-only" sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible. permalink fedilink source parent hideshow 1 child comment replies: [–] cupcakezealot@piefed.blahaj.zone 2 points 3 months ago ooh i'm sorry i didn't catch that <3 permalink fedilink source parent
[–] MonkderVierte@lemmy.zip 1 point 3 months ago* (1 child) Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big "js-only" sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible. permalink fedilink source parent hideshow 1 child comment replies: [–] cupcakezealot@piefed.blahaj.zone 2 points 3 months ago ooh i'm sorry i didn't catch that <3 permalink fedilink source parent
[–] cupcakezealot@piefed.blahaj.zone 2 points 3 months ago ooh i'm sorry i didn't catch that <3 permalink fedilink source parent