5
Anti SEO? (slrpnk.net)
submitted 11 months ago by to c/meta@slrpnk.net
4 comments fedilink hide all child comments
 

I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 3 points 11 months ago (3 children)

It's any day now I think, EU pension funds are moving out https://www.removepaywall.com/search?url=https%3A%2F%2Fwww.ft.com%2Fcontent%2F9d90d557-48e5-4f4b-a927-88071cef8ea9

Would you be up for re-enabling Google indexing? It is crappy, but still..

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 11 months ago (2 children)

    Not very motivated, but I can look into it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 11 months ago (1 child)

    I think it would just be

    User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 11 months ago*

    Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady.

    But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days.

  • permalink
  • fedilink
  • source
  • parent