5
Anti SEO? (slrpnk.net)
submitted 11 months ago by to c/meta@slrpnk.net
4 comments fedilink hide all child comments
 

I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

all 10 comments

sorted by: hot top controversial new old
[–] 8 points 11 months ago* (4 children)

At this point we try to block pretty much everything even remotely related to AI companies.

Soon we will probably have to block Chrome browsers when they start to use them to scrape websites without their users knowing (yes that is why AI companies started to make their own browsers and Mozilla is planning the same proudly proclaiming how "stealthy" they can be with that.).

Google search results have become so useless that I see little point left trying to accomodate their search bot 🤷

Yes I am bitter and can't wait for the AI bubble to pop.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S] 3 points 11 months ago (3 children)

    It's any day now I think, EU pension funds are moving out https://www.removepaywall.com/search?url=https%3A%2F%2Fwww.ft.com%2Fcontent%2F9d90d557-48e5-4f4b-a927-88071cef8ea9

    Would you be up for re-enabling Google indexing? It is crappy, but still..

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 11 months ago (2 children)

    Not very motivated, but I can look into it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 11 months ago (1 child)

    I think it would just be

    User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 11 months ago*

    Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady.

    But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 2 points 11 months ago

    There has and IIRC it's to help prevent scraping.

  • permalink
  • fedilink
  • source
    •
    [+] 1 point 11 months ago (2 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 11 months ago* (last edited 11 months ago) (1 child)

    In the bottom

    User-agent: *
Disallow: /

    Blocks all scrapers and superseeds the text above AFAIK

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • load more comments (-1 replies)