you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago

If there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt

Have a RSL license? here you go scrape the content.

Trying to freeload? anubis time.

  • source