Selfhosted

63,656 readers

5,234 active

545 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

moderation log

moderators

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

How to combat large amounts of Ai scrapers (sh.itjust.works)

submitted 1 year ago by DrunkAnRoot@sh.itjust.works to c/selfhosted@lemmy.world

53 comments fedilink hide all child comments

everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

you are viewing a single comment's thread
view the rest of the comments

[–] fubarx@lemmy.world 11 points 1 year ago (2 children)

If nginx, here's an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx

If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/

permalink

fedilink

source

hideshow 2 child comments

[–] ikidd@lemmy.world 3 points 1 year ago*

I wonder why that RoboNope doesn't just make a fail2ban entry for anything that accesses a disallowed url and drop them entirely.

Actually this look like it would do something similiar, then dumps them to fail2ban after the re-access the honeypot page too many times: https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/

permalink

fedilink

source

parent

[–] DrunkAnRoot@sh.itjust.works [S] 2 points 1 year ago

ill check robonope out seems promising

permalink

fedilink

source

parent