550
The Open-Source Software Saving the Internet From AI Bot Scrapers (www.404media.co)
submitted 1 year ago by to c/opensource@lemmy.ml
98 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 9 points 1 year ago* (5 children)

It is basically instantaneous on my 12 year old Keppler GPU Linux Box. It is substantially less impactful on the environment than AI tar pits and other deterrents. The Cryptography happening is something almost all browsers from the last 10 years can do natively that Scrapers have to be individually programmed to do. Making it several orders of magnitude beyond impractical for every single corporate bot to be repurposed for. Only to then be rendered moot, because it's an open-source project that someone will just update the cryptographic algorithm for. These posts contain links to articles, if you read them you might answer some of your own questions and have more to contribute to the conversation.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago (4 children)

    It is basically instantaneous on my 12 year old Keppler GPU Linux Box.

    It depends on what the website admin sets, but I've had checks take more than 20 seconds on my reasonably modern phone. And as scrapers get more ruthless, that difficulty setting will have to go up.

    The Cryptography happening is something almost all browsers from the last 10 years can do natively that Scrapers have to be individually programmed to do. Making it several orders of magnitude beyond impractical for every single corporate bot to be repurposed for.

    At best these browsers are going to have some efficient CPU implementation. Scrapers can send these challenges off to dedicated GPU farms or even FPGAs, which are an order of magnitude faster and more efficient. This is also not complex, a team of engineers could set this up in a few days.

    Only to then be rendered moot, because it's an open-source project that someone will just update the cryptographic algorithm for.

    There might be something in changing to a better, GPU resistant algorithm like argon2, but browsers don't support those natively so you would rely on an even less efficient implementation in js or wasm. Quickly changing details of the algorithm in a game of whack-a-mole could work to an extent, but that would turn this into an arms race. And the scrapers can afford far more development time than the maintainers of Anubis.

    These posts contain links to articles, if you read them you might answer some of your own questions and have more to contribute to the conversation.

    This is very condescending. I would prefer if you would just engage with my arguments.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago

    How will Anubis attack if browsers start acting like manual scrapers used by AI companies to collect information?

    Because OpenAI is planning to release an AI-powered browser, what happens if it ends up using it as another way to collect information?

    Blocking all Chromium browsers, I don't think, is a good idea.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 1 year ago

    Scrapers can send these challenges off to dedicated GPU farms or even FPGAs, which are an order of magnitude faster and more efficient.

    Lets assume for the sake of argument, an AI scraper company actually attempted this. They don't, but lets assume it anyway.

    The next Anubis release could include (for example), SHA256 instead of SHA1. This would be a simple, and basically transparent update for admins and end users. The AI company that invested into offloading the PoW to somewhere more efficient now has to spend significantly more resources changing their implementation than what it took for the devs and users of Anubis.

    Yes, it technically remains a game of "cat and mouse", but heavily stacked against the cat. One step for Anubis is 2000 steps for a company reimplementing its client in more efficient hardware. Most of the Anubis changes can even be done without impacting the end users at all. That's a game AI companies aren't willing to play, because they've basically already lost. It doesn't really matter how "efficient" the implementation is, if it can be rendered unusable by a small Anubis update.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 1 year ago* (last edited 1 year ago) (1 child)

    At best these browsers are going to have some efficient CPU implementation.

    Means absolutely nothing in context to what I said, or any information contained in this article. Does not relate to anything I originally replied to.

    Scrapers can send these challenges off to dedicated GPU farms or even FPGAs, which are an order of magnitude faster and more efficient.

    Not what's happening here. Be Serious.

    I would prefer if you would just engage with my arguments.

    I did, your arguments are bad and you're being intellectually disingenuous.

    This is very condescending.

    Yeah, that's the point. Very Astute

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •