It also doesn’t function without JavaScript. If you’re security or privacy conscious chances are not zero that you have JS disabled, in which case this presents a roadblock.

On the flip side of things, if you are a creator and you’d prefer to not make use of JS (there’s dozens of us) then forcing people to go through a JS “security check” feels kind of shit. The alternative is to just take the hammering, and that feels just as bad.

No hate on Anubis. Quite the opposite, really. It just sucks that we need it.

Scarcity is what powers this type of challenge: you have to prove you spent a certain amount of electricity in exchange for access to the site, and because electricity isn't free, this imposes a dollar cost on bots.

You could skip the detour through hashes/electricity and do something with a proof-of-stake cryptocurrency, and just pay for access. The site owner actually gets compensated instead of burning dead dinosaurs.

Obviously there are practical roadblocks to this today that a JavaScript proof-of-work challenge doesn't face, but longer term...

We have memory hard cryptographic functions

Did i forgot to mention it doesnt work without js that i keep disabled

Great article, but I disagree about WAFs.

Try to secure a nonprofit's web infrastructure with as 1 IT guy and no budget for devs or security.

It would be nice if we could update servers constantly and patch unmaintained code, but sometimes you just need to front it with something that plugs those holes until you have the capacity to do updates.

But 100% the WAF should be run locally, not a MiTM from evil US corp in bed with DHS.

Its always code forges and wikis that are effected by this because the scrapers spider down into every commit or edit in your entire history, then come back the next day and check every “page” again to see if any changed. Consider just blocking pages that are commit history at your reverse proxy.

This is the first time I've ever seen it misspelled like that. It's 'terabyte/terabytes'. 1,024 GBs worth of data.

Nice ads people! Good job!

It's a palette of 10 colours. I would guess it uses an indexed colorspace, reducing the size to a minimum.
edit: 28 KB on disk

Afaik, you can set it up not to have any image, or have any other one.

This is the way. I also have rules for hits to url, without a referer, that should never be hit without a referer, with some threshold to account for a user hitting F5. Plus a whitelist of real users (ones that got a 200 on a login endpoint). Mostly the Huawei and Tencent crawlers have fake user agents and no referer. Another thing crawlers don't do is caching. A user would never download that same .js file 100s of times in a hour, all their devices' browsers would have cached it. There's quite a lot of these kinds of patterns that can be used to block bots. Just takes watching the logs a bit to spot them.

Then there's ratelimiting and banning ip's that hit the ratelimit regularly. Use nginx as a reverse proxy, set rate limits for URLs where it makes sense, with some burst set, ban IPs that got rate-limited more than x times in the past y hours based on the rate limit message in the nginx error.log. Might need some fine tuning/tweaking to get the thresholds right but can catch some very spammy bots. Doesn't help with those that just crawl from 100s of ips but only use each ip once every hour, though.

Ban based on the bot user agents, for those that set it. Sure, theoretically robots.txt should be the way to deal with that, for well behaved crawlers, but if it's your homelab and you just don't want any crawlers, might as well just block those in the firewall the first time you see them.

Downloading abuse ip lists nightly and banning those, that's around 60k abusive ip's gone. At that point you probably need to use nftables directly though instead of iptables or going through ufw, for the sets, as having 60k rules would be a bad idea.

there's lists of all datacenter ip ranges out there, so you could block as well, though that's a pretty nuclear option, so better make sure traffic you want is whitelisted. E.g. for lemmy, you can get a list of the ips of all other instances nightly, so you don't accidentally block them. Lemmy traffic is very spammy…

there's so much that can be done with f2b and a bit of scripting/writing filters

You just described what wazuh does ootb

Hi, there are pre-made ipset lists also, ex: https://github.com/ktsaou/blocklist-ipsets

I've never had any issues on my phone using Fennec or Firefox. I don't have many addons installed apart from uBlock Origin. I wouldn't be surprised if some privacy addons cause issues with Anubis though.

Just imagine my pain on my phone. Js disabled, and takes a year to complete☠️ 

And on private tab, have to go through every time

all of the issues listed are closed so any recent version is fine.

also, you probably don't need to deploy this unless you have a problem with bots.

Security issues are always a concern the question is how much. Looking at it they seem to at most be ways to circumvent the Anubis redirect system to get to your page using very specific exploits. These are marked as m low to moderate priority and I do not see anything that implies like system level access which is the big concern. Obviously do what you feel is best but IMO its not worth sweating about. Nice thing about open source projects is that anyone can look through and fix, if this gets more popular you can expect bug bounties and professional pen testing submissions.

This isn't really a security issue as much as it is a DDOS issue.

Imagine you own a brick and mortar store. And periodically one thousand fucking people sprint into your store and start recording the UPCs on all the products, knocking over every product in the store along the way. They don't buy anything, they're exclusively there to collect information from your store which they can use to grift investors and burn precious resources, and if they fuck your shit up in the process, that's your problem.

This bot just sits at the door and ensures the people coming in there are actually shoppers interested in the content of some items of your store.

The front page of the web site is excellent. It describes what it does, and it does its feature set in quick, simple terms.

I can't tell you how many times I've gone to a website for some open-source software and had no idea what it was or how it was trying to do it. They often dive deep into the 300 different ways of installing it, tell you what the current version is and what features it has over the last version, but often they just assume you know the basics.

there's a fork that has non-js checks. I don't remember the name but maybe that's what should be made more known

I kinda sucks how AI scrapers make websites inaccessible to everyone 🙄

Something that hasn't been mentioned much in discussions about Anubis is that it has a graded tier system of how sketchy a client is and changing the kind of challenge based on a a weighted priority system.

The default bot policies it comes with has it so squeaky clean regular clients are passed through, then only slightly weighted clients/IPs get the metarefresh, then its when you get to moderate-suspicion level that JavaScript Proof of Work kicks. The bot policy and weight triggers for these levels, challenge action, and duration of clients validity are all configurable.

It seems to me that the sites who heavy hand the proof of work for every client with validity that only last every 5 minutes are the ones who are giving Anubis a bad wrap. The default bot policy settings Anubis comes with dont trigger PoW on the regular Firefox android clients ive tried including hardened ironfox. meanwhile other sites show the finger wag every connection no matter what.

Its understandable why some choose strict policies but they give the impression this is the only way it should be done which Is overkill. I'm glad theres config options to mitigate impact normal user experience.

POW is a far higher cost on your actual users than the bots.

That sentence tells me that you either don't understand or consciously ignore the purpose of Anubis. It's not to punish the scrapers, or to block access to the website's content. It is to reduce the load on the web server when it is flooded by scraper requests. Bots running headless Chrome can easily solve the challenge, but every second a client is working on the challenge is a second that the web server doesn't have to waste CPU cycles on serving clankers.

POW is an inconvenience to users. The flood of scrapers is an existential threat to independent websites. And there is a simple fact that you conveniently ignored: it fucking works.

Hashcash works great, what are you going on about?