▲ 5 ▼ Anti SEO? (slrpnk.net) submitted 11 months ago by sam_uk@slrpnk.net to c/meta@slrpnk.net 4 comments fedilink hide all child comments I wondered about the robots.txt I can see the case for it, I could also see the case for allowing at least Google to index the site. Has there been some discussion about this previously?
[–] poVoq@slrpnk.net 2 points 11 months ago (2 children) Not very motivated, but I can look into it. permalink fedilink source parent hideshow 2 child comments replies: [–] sam_uk@slrpnk.net [S] 2 points 11 months ago (1 child) I think it would just be User-agent: * Disallow: / User-agent: Googlebot Allow: / permalink fedilink source parent hideshow 1 child comment replies: [–] poVoq@slrpnk.net 2 points 11 months ago* Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady. But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days. permalink fedilink source parent
[–] sam_uk@slrpnk.net [S] 2 points 11 months ago (1 child) I think it would just be User-agent: * Disallow: / User-agent: Googlebot Allow: / permalink fedilink source parent hideshow 1 child comment replies: [–] poVoq@slrpnk.net 2 points 11 months ago* Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady. But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days. permalink fedilink source parent
[–] poVoq@slrpnk.net 2 points 11 months ago* Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady. But honestly, I highly doubt we will get much traffic from Google search. It's completely gone to shit these days. permalink fedilink source parent