▲ 581 ▼ New Executive Order:AI must agree on the Administration views on Sex,Race, cant mention what they deem to be Critical Race Theory,Unconscious Bias,Intersectionality,Systemic Racism or "Transgenderism (www.whitehouse.gov) submitted 1 year ago by FenrirIII@lemmy.world to c/technology@lemmy.world 100 comments fedilink hide all child comments cross-posted from: https://mander.xyz/post/34629331 cross-posted from: https://programming.dev/post/34472919 Reddit. Hackernews.
[–] ohshit604@sh.itjust.works 4 points 1 year ago (1 child) If your reverse proxy is Traefik I would suggest This plugin which pulls the robot.txt from This GitHub repository. I honestly should’ve setup a robots.txt a long time ago. permalink fedilink source parent hideshow 1 child comment replies: [–] xthexder@l.sw0.com 5 points 1 year ago Unfortunately robots.txt only stops the well behaved scrapers. Even with disallow all, you'll still get loads of bots. Setting up the web server to block those user agents would work a bit better, but even then there's bots out there crawling using regular browser user agents. permalink fedilink source parent
[–] xthexder@l.sw0.com 5 points 1 year ago Unfortunately robots.txt only stops the well behaved scrapers. Even with disallow all, you'll still get loads of bots. Setting up the web server to block those user agents would work a bit better, but even then there's bots out there crawling using regular browser user agents. permalink fedilink source parent