▲ 550 ▼ The Open-Source Software Saving the Internet From AI Bot Scrapers (www.404media.co) submitted 1 year ago by fattyfoods@feddit.nl to c/opensource@lemmy.ml 98 comments fedilink hide all child comments
[–] bdonvr@thelemmy.club 25 points 1 year ago (12 children) Ooh can this work with Lemmy without affecting federation? permalink fedilink source hideshow 12 child comments replies: [–] beyond@linkage.ds8.zone 32 points 1 year ago (2 children) Yes. Source: I use it on my instance and federation works fine permalink fedilink source parent hideshow 2 child comments replies: [–] bdonvr@thelemmy.club 16 points 1 year ago (1 child) Thanks. Anything special configuring it? permalink fedilink source parent hideshow 1 child comment replies: [–] beyond@linkage.ds8.zone 20 points 1 year ago* I keep my server config in a public git repo, but I don't think you have to do anything really special to make it work with lemmy. Since I use Traefik I followed the guide for setting up Anubis with Traefik. I don't expect to run into issues as Anubis specifically looks for user-agent strings that appear like human users (i.e. they contain the word "Mozilla" as most graphical web browsers do) any request clearly coming from a bot that identifies itself is left alone, and lemmy identifies itself as "Lemmy/{version} +{hostname}" in requests. permalink fedilink source parent [–] deadcade@lemmy.deadca.de 10 points 1 year ago (1 child) "Yes", for any bits the user sees. The frontend UI can be behind Anubis without issues. The API, including both user and federation, cannot. We expect "bots" to use an API, so you can't put human verification in front of it. These "bots* also include applications that aren't aware of Anubis, or unable to pass it, like all third party Lemmy apps. That does stop almost all generic AI scraping, though it does not prevent targeted abuse. permalink fedilink source parent hideshow 1 child comment replies: [–] beyond@linkage.ds8.zone 3 points 1 year ago The API, including both user and federation, cannot. This is theoretically an issue however in practice Anubis only weighs requests that appear to come from a browser: https://anubis.techaro.lol/docs/design/how-anubis-works I just tested my instance with Jerboa and it seems to work just fine. permalink fedilink source parent [–] interdimensionalmeme@lemmy.ml 8 points 1 year ago (2 children) Yes, it would make lemmy as unsearchable as discord. Instead of unsearchable as pinterest. permalink fedilink source parent hideshow 2 child comments replies: [–] bdonvr@thelemmy.club 3 points 1 year ago (1 child) That's not true, search indexer bots should be allowed through from what I read here. permalink fedilink source parent hideshow 1 child comment replies: [–] interdimensionalmeme@lemmy.ml 8 points 1 year ago If you allow my searchxng search scraper then an AI scraper is indistinguishable. If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot. permalink fedilink source parent [–] infinitesunrise@slrpnk.net 6 points 1 year ago Yeah, it's already deployed on slrpnk.net. I see it momentarily every time I load the site. permalink fedilink source parent [–] seang96@spgrn.com 4 points 1 year ago As long as its not configured improperly. When forgejo devs added it it broke downloading images with Kubernetes for a moment. Basically would need to make sure user agent header for federation is allowed. permalink fedilink source parent [–] Resonosity@lemmy.dbzer0.com 2 points 1 year ago (1 child) To be honest, I need to ask my admin about that! permalink fedilink source parent hideshow 1 child comment replies: [–] fxomt@lemmy.dbzer0.com 3 points 1 year ago We don't use anubis but we use Iocain, see /0 for the announcement post permalink fedilink source parent
[–] beyond@linkage.ds8.zone 32 points 1 year ago (2 children) Yes. Source: I use it on my instance and federation works fine permalink fedilink source parent hideshow 2 child comments replies: [–] bdonvr@thelemmy.club 16 points 1 year ago (1 child) Thanks. Anything special configuring it? permalink fedilink source parent hideshow 1 child comment replies: [–] beyond@linkage.ds8.zone 20 points 1 year ago* I keep my server config in a public git repo, but I don't think you have to do anything really special to make it work with lemmy. Since I use Traefik I followed the guide for setting up Anubis with Traefik. I don't expect to run into issues as Anubis specifically looks for user-agent strings that appear like human users (i.e. they contain the word "Mozilla" as most graphical web browsers do) any request clearly coming from a bot that identifies itself is left alone, and lemmy identifies itself as "Lemmy/{version} +{hostname}" in requests. permalink fedilink source parent
[–] bdonvr@thelemmy.club 16 points 1 year ago (1 child) Thanks. Anything special configuring it? permalink fedilink source parent hideshow 1 child comment replies: [–] beyond@linkage.ds8.zone 20 points 1 year ago* I keep my server config in a public git repo, but I don't think you have to do anything really special to make it work with lemmy. Since I use Traefik I followed the guide for setting up Anubis with Traefik. I don't expect to run into issues as Anubis specifically looks for user-agent strings that appear like human users (i.e. they contain the word "Mozilla" as most graphical web browsers do) any request clearly coming from a bot that identifies itself is left alone, and lemmy identifies itself as "Lemmy/{version} +{hostname}" in requests. permalink fedilink source parent
[–] beyond@linkage.ds8.zone 20 points 1 year ago* I keep my server config in a public git repo, but I don't think you have to do anything really special to make it work with lemmy. Since I use Traefik I followed the guide for setting up Anubis with Traefik. I don't expect to run into issues as Anubis specifically looks for user-agent strings that appear like human users (i.e. they contain the word "Mozilla" as most graphical web browsers do) any request clearly coming from a bot that identifies itself is left alone, and lemmy identifies itself as "Lemmy/{version} +{hostname}" in requests. permalink fedilink source parent
[–] deadcade@lemmy.deadca.de 10 points 1 year ago (1 child) "Yes", for any bits the user sees. The frontend UI can be behind Anubis without issues. The API, including both user and federation, cannot. We expect "bots" to use an API, so you can't put human verification in front of it. These "bots* also include applications that aren't aware of Anubis, or unable to pass it, like all third party Lemmy apps. That does stop almost all generic AI scraping, though it does not prevent targeted abuse. permalink fedilink source parent hideshow 1 child comment replies: [–] beyond@linkage.ds8.zone 3 points 1 year ago The API, including both user and federation, cannot. This is theoretically an issue however in practice Anubis only weighs requests that appear to come from a browser: https://anubis.techaro.lol/docs/design/how-anubis-works I just tested my instance with Jerboa and it seems to work just fine. permalink fedilink source parent
[–] beyond@linkage.ds8.zone 3 points 1 year ago The API, including both user and federation, cannot. This is theoretically an issue however in practice Anubis only weighs requests that appear to come from a browser: https://anubis.techaro.lol/docs/design/how-anubis-works I just tested my instance with Jerboa and it seems to work just fine. permalink fedilink source parent
[–] interdimensionalmeme@lemmy.ml 8 points 1 year ago (2 children) Yes, it would make lemmy as unsearchable as discord. Instead of unsearchable as pinterest. permalink fedilink source parent hideshow 2 child comments replies: [–] bdonvr@thelemmy.club 3 points 1 year ago (1 child) That's not true, search indexer bots should be allowed through from what I read here. permalink fedilink source parent hideshow 1 child comment replies: [–] interdimensionalmeme@lemmy.ml 8 points 1 year ago If you allow my searchxng search scraper then an AI scraper is indistinguishable. If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot. permalink fedilink source parent
[–] bdonvr@thelemmy.club 3 points 1 year ago (1 child) That's not true, search indexer bots should be allowed through from what I read here. permalink fedilink source parent hideshow 1 child comment replies: [–] interdimensionalmeme@lemmy.ml 8 points 1 year ago If you allow my searchxng search scraper then an AI scraper is indistinguishable. If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot. permalink fedilink source parent
[–] interdimensionalmeme@lemmy.ml 8 points 1 year ago If you allow my searchxng search scraper then an AI scraper is indistinguishable. If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot. permalink fedilink source parent
[–] infinitesunrise@slrpnk.net 6 points 1 year ago Yeah, it's already deployed on slrpnk.net. I see it momentarily every time I load the site. permalink fedilink source parent
[–] seang96@spgrn.com 4 points 1 year ago As long as its not configured improperly. When forgejo devs added it it broke downloading images with Kubernetes for a moment. Basically would need to make sure user agent header for federation is allowed. permalink fedilink source parent
[–] Resonosity@lemmy.dbzer0.com 2 points 1 year ago (1 child) To be honest, I need to ask my admin about that! permalink fedilink source parent hideshow 1 child comment replies: [–] fxomt@lemmy.dbzer0.com 3 points 1 year ago We don't use anubis but we use Iocain, see /0 for the announcement post permalink fedilink source parent
[–] fxomt@lemmy.dbzer0.com 3 points 1 year ago We don't use anubis but we use Iocain, see /0 for the announcement post permalink fedilink source parent