I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 11 months ago* (4 children)

At this point we try to block pretty much everything even remotely related to AI companies.

Soon we will probably have to block Chrome browsers when they start to use them to scrape websites without their users knowing (yes that is why AI companies started to make their own browsers and Mozilla is planning the same proudly proclaiming how "stealthy" they can be with that.).

Google search results have become so useless that I see little point left trying to accomodate their search bot 🤷

Yes I am bitter and can't wait for the AI bubble to pop.

  • source
  • hideshow 4 child comments
  • [–] [S] 3 points 11 months ago (3 children)

    It's any day now I think, EU pension funds are moving out https://www.removepaywall.com/search?url=https%3A%2F%2Fwww.ft.com%2Fcontent%2F9d90d557-48e5-4f4b-a927-88071cef8ea9

    Would you be up for re-enabling Google indexing? It is crappy, but still..

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 11 months ago (2 children)

    Not very motivated, but I can look into it.

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 2 points 11 months ago (1 child)

    I think it would just be

    User-agent: *
    Disallow: /
    User-agent: Googlebot
    Allow: /
    
  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 11 months ago*

    Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady.

    But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days.

  • source
  • parent