▲ 5 ▼ Anti SEO? (slrpnk.net) submitted 11 months ago by sam_uk@slrpnk.net to c/meta@slrpnk.net 4 comments fedilink hide all child comments I wondered about the robots.txt I can see the case for it, I could also see the case for allowing at least Google to index the site. Has there been some discussion about this previously?
[–] Nemo@slrpnk.net 2 points 11 months ago There has and IIRC it's to help prevent scraping. permalink fedilink source