1
 
 

A log line gives you a token. What you need in order to decide anything is the organisation behind it, and that mapping is genuinely annoying to assemble: one operator often runs eight or ten differently named crawlers with different purposes and different opt-outs, and the names change with rebrands.

So this is one page per operator - 74 of them, covering 150 crawlers - and each page carries:

  • every token that operator runs, with its category: AI training, AI search, user-triggered fetch, classic search index, SEO, archive, liveness check
  • the verification method the operator itself publishes - reverse DNS where they document a convention, a published IP range list where they do not. 1987 IPv4 and 1062 IPv6 prefixes from 15 such endpoints are mirrored here, re-fetched every six hours, with a change feed for when a range moves
  • the opt-out tokens that are separate from the crawler token, which is where most robots.txt files are wrong: Google-Extended and Applebot-Extended control training, not crawling, and blocking the crawler token instead removes you from search while leaving the training question untouched
  • what blocking each one costs you, in one plain sentence per crawler

Useful for the case this community meets most: an unfamiliar user-agent in an access log, and the question of whether it is who it says it is before you write a firewall rule about it.

Static files, no key, no rate limit, CORS open. Data CC0, tooling MIT.

https://www.pathwren.workers.dev/c/piefed/operator/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

2
 
 

A/I stands for Autistici/Inventati.

A/I (Autistici/Inventati, pronounced [au’tistiʧi]-[iŋ’vɛntati], or [iŋvɛn’tati]) was born in 2001 from an encounter of individuals and collectives of the autonomous anticapitalist movement interested in technology and active in the digital rights struggle. We believe that this world is far from being the best world possible. We react by providing a platform and tools for digital self-defense addressing the need of free communication for activists and other individuals.


The day we discovered we had been designated a global terrorist organization, we promised that we would try to resist as long as this was possible, that we would not back down as long as we could see options to stand our ground.

We cannot engage in a fight or expose ourselves to manipulations that threaten the lives of these people and their loved ones, leaving them at the mercy of autocrats, fascists and agencies that take extra-legal courses of action.

What happened? First their main domain, autistici.org, was blocked globally by the US entity that provides the ".org" TLD. More recently their bank shut down their funds because of caving in to pressure that ultimately comes from the same source.

Full context/statement about the domain shutdown:
https://www.inventati.org/campaign/press

I also made some posts about this earlier. You can probably guess that I've been relying on their services, for a long time. And not even for particularly political reasons - they give email accounts to people who care for privacy and anonymity. Or used to.

They announce this new site:
https://keepitfree.ai/announcements/announcing-https/keepitfree.ai/


Personally I'm just shocked how US policy reaches and impacts my normal peaceful European life.

Fuck the US government. Being AntiFa is not terrorism. Being against AntiFa means you're Fa.


PS: Sorry for posting this to several world news communities. I feel this topic should shake the Fediverse.
Remember:
This can happen to any ".org" domain, and possibly many other TLDs that are controled by entities that are prone to give into US bullying. Or your bank will cave to US pressure.

3
 
 

cross-posted from: https://piefed.social/c/fediverse/p/1569729/easily-set-up-your-piefed-instance-using-yunohost

After 5 months of chipping away at it, PieFed is now an installable 'app' in the Yunohost store! It's been there for a couple of weeks actually but until this weekend it had a scary red exclamation mark because some automated tests hadn't ran yet. But that's gone now so I feel confident about recommending it to others.

Yunohost is a linux distro for servers that has a web gui for installing and managing services, that takes all the hassle out of self-hosting. How to get started with Yunohost.

@squirrel@piefed.kobel.fyi and @michael@piefed.chrisco.me have had good success setting up their instances already: https://piefed.social/c/piefed_meta/p/1561141/thanks-to-rimu-ericgaspar-and-tituspijean-yunohost-has-a-working-piefed-setup

4
 
 

I've been putting together a list of independent news sources lately. Here's a decent list that someone posted on their newsletter: https://www.thehandbasket.co/p/independent-journalism-gift-guide

5
 
 

As Snowden told us, video and audio recording capabilities of your devices are NSA spying vectors. OSS/Linux is a safeguard against such capabilities. The massive datacenter investments in US will be used to classify us all into a patriotic (for Israel)/Oligarchist social credit score, and every mega tech company can increase profits through NSA cooperation, and are legally obligated to cooperate with all government orders.

Speech to text and speech automation are useful tech, though always listening state sponsored terrorists is a non-NSA targeted path for sweeping future social credit classifications of your past life.

Some small LLMs that can be used for speech to text: https://modal.com/blog/open-source-stt

6
 
 

cross-posted from: https://slrpnk.net/post/29975018

Fedfree is a website aimed at teaching people how to run their own servers, of various kinds, on libre operating systems e.g. Linux and BSD. It aims to do this, using libre software exclusively, teaching people about the importance of libre software and hardware as it pertains to freedom; the right to use, study, adapt, share. The right to read. Universal access to knowledge... education. Education is the goal.

Fedfree's mission is to bring back the real internet to normal people, the one where you can have your own unique voice on the internet, without plugging into the hive mind that is websites like twitter or youtube.

Most of the internet's problems exist, precisely because of modern centralised providers holding us back from true innovation.

The goal is to spread libre software ideology, while providing a practical means for people to know how to conduct themselves, such as:

  • Advice about how to start your own projects
  • How to get into computer science, electronics and other computer-related fields... with a view towards libre ideology
  • How to run and maintain your own infrastructure, free from interference, completely hardened against intrusion
  • Link to resources, covering many different topics. For example, when to use FreeBSD vs OpenBSD or vice versa, or situations where Linux might be better, comparing various servers e.g. postfix vs opensmtpd
  • Other examples of links to resources could include: links to books about programming languages, networking concepts, from beginner level all the way to BOFH (Bastard Operator From Hell)

The real internet exists. Fedfree's mission is to teach you how to use it. Every part of it. To most people, it is hidden. Your ISP might put you behind a CGNAT for example, or outright ban you from opening ports; one of Fedfree's goals is to teach you how to set up various kinds of tunnel connections e.g. SSH port forwarding, PPP over L2TP, Wireguard/OpenVPN, etc.

The mentality behind Fedfree is that all the organisations out there, like SFC, GNU, EFF, FSF... April... all these organisations are good, but they can only do so much. We as libre software activists must organise, but how? First, we need infrastructure, our own infrastructure that we control, and we need a charter that defines our movement. By definition, the libre movement is loose and free, where people can do whatever they like, but most people today use centralised hosting services like Github, which means we have huge single points of failure.

7
 
 
8
 
 

A place to discuss the use of the internet with less corporate influence.

All things I see here are not about using the internet with less corporate influence, they are about how the internet with corporate influence is bad. While I agree, I thought this community would be more about showing examples of and promoting less-corporately-influenced platforms than doom news about the current corporate hellscape that I could already get from the many communities named Technology or something like that. I fully admit I am posting out of frustration, and am happy to unsubscribe with no ill will towards anybody if this community's intent is to post more about the problems present in a corporate-influenced internet. (And yes, "be the change you want to see," I get it. I used to post on !webrevival@lemm.ee and me existing in this community is me trying to find a substitute for that now that lemm.ee is dead.)

9
 
 

cross-posted from: https://programming.dev/post/37759200

Parents outraged as Meta uses photos of schoolgirls in ads targeting man: Instagram pictures of girls as young as 13 were posted to promote Threads site ‘as bait’, campaigner says

Comments

10
 
 

cross-posted from: https://programming.dev/post/37228898

  • Meta uses an array of influence tactics to try to shape the public narrative around kids and social media. The goal is to counter growing concerns that sites like Instagram can be harmful to teens’ health and safety.
  • Meta funds a collection of parent and child safety groups, including the National PTA, that go to bat for its initiatives involving kids. This gives a sheen of expert approval to the company’s efforts to keep young users engaged on its platforms.
  • Meta has also created something called the Trust, Transparency & Control Labs that publishes reports in support of its kid-focused products. Meta has at times framed these “labs” as a separate organization to regulators and others.
  • Meanwhile Meta has funded an array of academic research projects that foster a more benign view of Instagram, helping to support the company’s contention that academic research is inconclusive on the topic of social media’s impact.
11
 
 

A video essay on what we give up in exchange for the convenience that social media and algorithms provide.

12