[–] 2 points 1 week ago

A lot of article websites whitelist Google's crawler, right? Can't that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though

  • source
  • parent
  • context
  • view more: next ›