you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 10 months ago

Beware of dog signs also have no enforcement of law.

But apart from that, if your crawler ends up stuck in an endless loop, that's poor coding on your part. Human beings won't browse a static website endlessly, neither should a crawler

  • source
  • parent