5
Anti SEO? (slrpnk.net)
submitted 11 months ago by to c/meta@slrpnk.net
4 comments fedilink hide all child comments
 

I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

you are viewing a single comment's thread
view the rest of the comments
[+] 1 point 11 months ago (2 children)
[deleted]
  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 11 months ago* (last edited 11 months ago) (1 child)

    In the bottom

    User-agent: *
Disallow: /

    Blocks all scrapers and superseeds the text above AFAIK

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • load more comments (-1 replies)