▲ 138 ▼ Discord will soon let anyone find public server messages through Google (discord.com) submitted 1 week ago by beep@piefed.world to c/technology@lemmy.world 39 comments fedilink hide all child comments
[–] AppleMango@lemmy.world 3 points 1 week ago (3 children) Could whitelist specific search engine crawlers permalink fedilink source parent hideshow 3 child comments replies: [–] diaphragmwp@discuss.tchncs.de 3 points 1 week ago (2 children) The bots will pretend to be search engine crawlers. And not all have their own IP ranges. permalink fedilink source parent hideshow 2 child comments replies: [–] Sharks@pawb.social 3 points 1 week ago* (last edited 1 week ago) You can't impersonate search engine crawlers, because the server can verify that it's really a search engine crawler See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot permalink fedilink source parent [–] AppleMango@lemmy.world 2 points 1 week ago A lot of article websites whitelist Google's crawler, right? Can't that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though permalink fedilink source parent
[–] diaphragmwp@discuss.tchncs.de 3 points 1 week ago (2 children) The bots will pretend to be search engine crawlers. And not all have their own IP ranges. permalink fedilink source parent hideshow 2 child comments replies: [–] Sharks@pawb.social 3 points 1 week ago* (last edited 1 week ago) You can't impersonate search engine crawlers, because the server can verify that it's really a search engine crawler See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot permalink fedilink source parent [–] AppleMango@lemmy.world 2 points 1 week ago A lot of article websites whitelist Google's crawler, right? Can't that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though permalink fedilink source parent
[–] Sharks@pawb.social 3 points 1 week ago* (last edited 1 week ago) You can't impersonate search engine crawlers, because the server can verify that it's really a search engine crawler See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot permalink fedilink source parent
[–] AppleMango@lemmy.world 2 points 1 week ago A lot of article websites whitelist Google's crawler, right? Can't that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though permalink fedilink source parent