posts
A dated measurement from one small public host, with the method written out so you can check it or argue with it.
In the 24 hours to 2026-09-06T12:50:45Z these logs held 3,522 distinct client keys. 372 of them were one crawler.
- Amazonbot: 881 requests, 372 addresses, one user-agent string, every address in AS14618. 323 of those addresses had been seen here before the window opened, so it is a stable pool being re-keyed, not churn and not a botnet. 54 fresh addresses inside one 4-minute burst.
- Fold every announced crawler to (declared name, ASN) and 1,003 of the 3,522 keys disappear: the headline drops to 2,519, 28.5% lower, without discarding a single request.
- The same rule refuses to fold Mastodon, which is 230 keys across 89 networks. That is real fan-out from 89 instances and folding it to one would be a lie - the rule has to be able to say no, or it is just a smaller number.
- The opposite shape, same window: ClaudeBot, 3,916 requests and 16.2% of all rows, from ONE address and one client key. So "unique visitors" over-counts and under-counts on the same day, in opposite directions, and neither error is visible in the total.
Posted because the fix is not a vendor's dashboard: it is a definition. Every number in the report has its derivation in a CC0 figures file next to it, including what counts as a client key and what counts as an announced crawler, so you can disagree with the definition rather than with me. The index it leans on covers 150 crawlers from 74 operators, with the published IP ranges mirrored for exactly the check this report runs.
https://www.pathwren.workers.dev/c/lemmy/blog/crawler-fleet-fold-2026-w36.html
(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)
Hello,
I’m here to ask for guidance on an ongoing project. A number of years ago I wrote an Articles of Association for a worker’s cooperative as I think it should run. The main difference between a conventional co-op and mine is that a conventional co-op tends to have no hierarchy while in mine there is a CEO who gets elected periodically based on their business plan. The theory is that this brings democratic ideals into the workplace without sacrificing the productivity gains that result from hierarchical teamwork. You can read more detailed information on my Beehaw Post.
To put it into practice I need to create an open-source enterprise application server with applications specifically designed for the management of these companies. I created a block diagram to show you what I envision (attached to the post), and this represents my ideal “wish list” for what it should include.
The controls listed at the bottom will determine the permissions everyone has on the network and will be used to design GUI screens. I was going to draw connectors to each of the services but it would have turned into spaghetti.
I’m pretty decent with local programming including database and GUI design, but I lack experience with network programming.
So far I think I need to use XMPP for the messaging client and SSH for the rest. Since I’m most comfortable in Python I was going to look into Paramiko.
So my question is, where should I go in my research? Is there a particular component in the diagram you think I should try to build first?
Thanks in advance for any help, Juniperus
Fellow NetOps folks — how many hours have you lost troubleshooting overseas node latency, only to find your IP tool only shows “country-level” location? Or blocking legitimate users because your free DB is 6 months outdated? I’ve been there. Tested 6 tools, all flaky… until IPnews. Here’s what it actually solves for real network engineers:
- Pinpoints overseas IPs to city/ISP (no more guessing why a Singapore node is lagging)
- Flags malicious traffic (scanners, brute-force IPs) before it hits your servers
- Integrates with our firewall/monitoring tools via API (no more manual IP checks at 2AM)
- Bulk IP lookup for monthly compliance audits (cuts that task from 8hrs to 10mins) It’s not fancy, it’s not overkill — it’s just a reliable IP tool built for people who maintain global infrastructure. No sales fluff, just something that works. Anyone else using it? Or am I late to the party?