freedomPusher comments on We need a browser that masquerades as a crawler -- to disenshitify pollution in our search results by unreachable content

We need a browser that masquerades as a crawler -- to disenshitify pollution in our search results by unreachable content (sopuli.xyz)

submitted 8 months ago by freedomPusher@sopuli.xyz to c/disenshittify@lemmy.cafe

2 comments fedilink hide all child comments

cross-posted from !foss_requests@libretechni.ca

In light of this news, we need a browser that looks like a search engine crawler.

This would equalise the problem of websites giving preferential treatment to crawlers and lousy treatment to the rest.

My question is: assuming all hearders could mimick a crawler, would that be sufficient? Or do paywalls take IP address into account? And if so, would it work to subscribe to Google Cloud just to get an IP address in Google’s ranges and use that for crawling?

[–] freedomPusher@sopuli.xyz [S] 3 points 8 months ago*

Do you mean that a browser cannot fool a web server without using AI? Can you explain in more detail how the web service detects a client’s use of AI?

permalink

fedilink

source

parent