you are viewing a single comment's thread
view the rest of the comments
[–] 10 points 10 months ago* (2 children)

It's deliberately slow to load

That kinda defeats the goal of feeding AI as much garbage as possible. They will just fetch a page from a different site in that time, instead of spending cycles on this page. It's not like the crawler works strictly serially.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 38 points 10 months ago* (1 child)

    The idea is to protect own server from unnecessary loads. You're welcome to provide a faster AI tar pit, just mind that ultimately this is a waste of resources.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 8 points 10 months ago* (last edited 10 months ago)

    I'm guessing that Markov chains are pretty efficient computationally compared to AI training. Don't have a site currently, but I'd love to see a bot rip through hundreds of pages a minute.

  • source
  • parent