We need a browser that masquerades as a crawler -- to disenshitify pollution in our search results by unreachable content : disenshittify@lemmy.cafe

We need a browser that masquerades as a crawler -- to disenshitify pollution in our search results by unreachable content (sopuli.xyz)

submitted 8 months ago by freedomPusher@sopuli.xyz to c/disenshittify@lemmy.cafe

2 comments fedilink hide all child comments

cross-posted from !foss_requests@libretechni.ca

In light of this news, we need a browser that looks like a search engine crawler.

This would equalise the problem of websites giving preferential treatment to crawlers and lousy treatment to the rest.

My question is: assuming all hearders could mimick a crawler, would that be sufficient? Or do paywalls take IP address into account? And if so, would it work to subscribe to Google Cloud just to get an IP address in Google’s ranges and use that for crawling?

all 2 comments

sorted by: hot top controversial new old

[–] riskable@programming.dev -2 points 8 months ago (1 child)

You mean a browser with AI features built in? Sounds useful!

permalink

fedilink

source

hideshow 1 child comment

[–] freedomPusher@sopuli.xyz [S] 3 points 8 months ago*

Do you mean that a browser cannot fool a web server without using AI? Can you explain in more detail how the web service detects a client’s use of AI?

permalink

fedilink

source

parent