550
The Open-Source Software Saving the Internet From AI Bot Scrapers (www.404media.co)
submitted 1 year ago by to c/opensource@lemmy.ml
98 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago (2 children)

This would not be a problem if one bot scraped once, and the result was then mirrored to all on Big Tech's dime (cloudflare, tailscale) but since they are all competing now, they think their edge is going to be their own more better scraper setup and they won't share.

Maybe there should just be a web to torrent bridge sovtge data is pushed out once by the server and tge swarm does the heavy lifting as a cache.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    No, it'd still be a problem; every diff between commits is expensive to render to web, even if "only one company" is scraping it, "only one time". Many of these applications are designed for humans, not scrapers.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] -1 points 1 year ago

    If the rendering data for scraper was really the problem Then the solution is simple, just have downloadable dumps of the publicly available information That would be extremely efficient and cost fractions of pennies in monthly bandwidth Plus the data would be far more usable for whatever they are using it for.

    The problem is trying to have freely available data, but for the host to maintain the ability to leverage this data later.

    I don't think we can have both of these.

  • permalink
  • fedilink
  • source
  • parent