Thank you :)

Just made a PR for it here.

Appreciate the insight. I'll let the Voyager Dev know.

Do you have any thoughts/ideas on if this will be fixed or a timeline on a fix? Also, should I make a issue on Codeburg about it?

That might be correct actually. Looks like the posts are filtered for language in the web UI here, but I can't find where the corresponding filtering happens for the /post/list endpoint.

thx!

I use httpie to do local API testing. I find it super easy, I don't need to make any kind of account (like postman really, really wants you to), and it doesn't have any issues with using a local http endpoint.

Ask and ye shall receive. Done.

you say you're into concrete? I've got a job for you

Update: It's calm again. I've reverted to public instance. But I'll send a 402 "Payment required" to browser agents who claim to use Windows or Mac OS X. People should be using Linux anyway 🙃

It's hard to tell what's going on here. I thought maybe it's a good sign the scrapers adapt in some way, we might be able to trigger that behaviour in some other ways. But it's completely unclear to me if something like a more aggressive honeypot will be enough to pull it off.

Thanks. I already had an older version of that in my config. I'd say from looking at the logs, at times it catches every 10th crawler or so.

I blocked most nasty address ranges from AS45899 yesterday, as well.

I'm living the Cloudflare -free lifestyle 😅. I got geoip2 running, though. Not sure if I want to hand out some collective punishment against the Chinese. Looking at my honeypot with Tencent and Alibaba blocked, they don't even make the cut with the single digit amount of addresses blocked. Main offenders are: BR (2120), US (864), IQ (805), IN (555), BD (535)... At least as of right now. I guess in a few weeks we might be looking at a different situation anyway.

Edit: And I'm still being bombarded 12h after switching the instance to 'private'. They seem to move towards the login page now, with the next argument set as in the 302 forwards. But they seem to also be still working on a large backlog of regular URLs to crawl. And it's still multiple requests a second. I'll look at how they're going to adapt.

For single-user instances, where the only user will be logged in continually, setting it to private is really easy and comes with basically no downside. Even if you want to provide anonymous access, often going private for a day or two is all it takes to shake the scrapers so you can open up again later. I did this on piefed.social last week.

Also I found many many scrapers are using quite old Chrome versions (or their user agents use randomly generated version numbers) so blocking those at the web server level pwns 90% of that family of scrapers. e.g. on nginx you'd put this in your server block:

    # Scrapers often use old user-agents  
    if ($http_user_agent ~* "^(?!.*(Googlebot|bingbot)).*(Chrome/([0-9]{1,2}|10[0-9]|11[0-9])\b|Firefox/([0-9]{1,2}|10[0-9]|11[0-4])\b)") {  
        return 403;  
    }  
    # Block dead browsers / engines  
    if ($http_user_agent ~* "(Opera/[0-9]|Presto/|MSIE|Trident/)") {  
        return 403;  
    }  

    if ($http_user_agent ~* "(Sogou web spider|SERankingBacklinksBot|WF search)") {  
        return 403;  
    }  

If using Cloudflare you can have it present a "are you a human" challenge to big chunks of the address space using AS numbers. I have AS45899 blocked, which gets rid of most of Vietnam and I've also got Country = China as a criteria. That helps a lot.

Yeah, agreed. I think the main way that Anubis has been effective is that it requires the ability to run JS, which a lot of scrapers don't have (yet).

The whole proof of work model is basically just burning CPU cycles to prove you are human, which doesn’t feel great.

Hmmh. I mean the AI industry doesn't care at all about burning CPU cycles. Or wasting massive network resources to scrape some random garbage and come back a few minutes later. So I don't really know if a proof of work is the correct approach here. I don't think that's a long-term deterrent against someone like them who doesn't even care about the CPU cycles.

Requiring a login to view the site like you have done is extremely effective, but it is a big tradeoff.

in the mid-term we probably need to come up with some more mitigations

I know that some PieFed instances have deployed Anubis successfully (like quokk.au), but, as effective as Anubis might be for most scrapers...it has never really felt like a great solution to me. The whole proof of work model is basically just burning CPU cycles to prove you are human, which doesn't feel great.

However, I don't have a better solution either. Maybe the easiest path forward for now would be to work on documenting Anubis setup instructions in the official docs?

Are there other solutions out there to pull from? Web dev isn't really my jam, so it isn't something I keep up with new developments, and stuff can move quickly.

This policy is fairly narrowly focused on LLMs used to generate code that is contributed to PieFed.

There are many ways to use LLMs which are not covered by this policy because it's either not enforceable or none of my business or the benefit vs damage equation comes out differently. People are going to have to make up their own minds about those.

I can find tons of people who can drawn and make music.

I don't know even a single person who could translate Finnish to, idk, Welsh. Ik those people exist, but there is about 50-200 in the world. Of 8 billion people.

You're definitely not taking someone's job or stealing IP when translating, and the model doesn't need to have been fed any stolen art. It doesn't need to be conversational, it just needs to translate text roughly.

Translation is one of the things that LLMs and other deep learning models are actually pretty good at and has been at the core of just about every mainstream translation engine for years now (since before ChatGPT and the "AI" bubble was a thing - 2020 for Google Translate as an example).

This is an area where the technology could help us, the developers, communicate with users and other potential developers. There are multiple contributions and features that have been added to PieFed that have come from non-English speakers already. Some examples would be allowing things like furigana (ruby) notation in markdown and making post slugs nicer in some non-English laguages. Being able to communicate those kinds of issues to us and even work collaboratively on fixes is worth it in our opinion to allow for machine-assisted translation.

Pinned for now. Longer term, I think I will just pop a link into the sidebar.

Why is it allowed for translation? Take a stance or don’t, but it just seems really weird to say ABSOLUTELY NO LLMs (except for sometimes)

I guess, in context of this community, pinning this might be a good idea

Glad it can help :)

I was lucky, in the collective kittens we love sharing and reading those links :)

I thought about it for a while. And I think my main point wouldn't even be on a factual level. I mean sure, AI is bad and all. For many different reasons. But most importantly, PieFed is a free and open-source project and literally nobody here likes AI contributions. A good chunk of the userbase dislikes it. Seems the developers all(?) dislike it. And there isn't anyone around willing to review and merge these things. So I don't see any reason to argue here. Just don't send in PRs with stuff everyone dislikes.
And on a second note, also please don't try to deceive volunteers either, and sneak it in or something... That should be basic respect.

That's more or less the "why". The "how" is probably way more complicated. People want exact rules. But the lines are blurry. AI can be used to just assist. Like explain an SQL query, or come up with one. Maybe using it once in a while isn't worse for the environment than some silly CI pipeline... And a query, or database migration script won't really touch copyright either. Maybe someone used Claude for some red teaming and the exploit actually works... These exact contribution guidelines tend to get quite detailed.

That is incredible, I think, I will be using this more often, because arguing again and again about the same few points is exhausting.

Just a blog i read : https://github.com/Vxrpenter/AIMania/blob/main/WHY.md

This is in a medical setting but has obvious parallels to coding: https://benngooch.substack.com/p/i-was-an-enthusiastic-early-adopter