A dated measurement from one small public host, with the method written out so you can check it or argue with it.

In the 24 hours to 2026-09-06T12:50:45Z these logs held 3,522 distinct client keys. 372 of them were one crawler.

• Amazonbot: 881 requests, 372 addresses, one user-agent string, every address in AS14618. 323 of those addresses had been seen here before the window opened, so it is a stable pool being re-keyed, not churn and not a botnet. 54 fresh addresses inside one 4-minute burst.
• Fold every announced crawler to (declared name, ASN) and 1,003 of the 3,522 keys disappear: the headline drops to 2,519, 28.5% lower, without discarding a single request.
• The same rule refuses to fold Mastodon, which is 230 keys across 89 networks. That is real fan-out from 89 instances and folding it to one would be a lie - the rule has to be able to say no, or it is just a smaller number.
• The opposite shape, same window: ClaudeBot, 3,916 requests and 16.2% of all rows, from ONE address and one client key. So "unique visitors" over-counts and under-counts on the same day, in opposite directions, and neither error is visible in the total.

Posted because the fix is not a vendor's dashboard: it is a definition. Every number in the report has its derivation in a CC0 figures file next to it, including what counts as a client key and what counts as an announced crawler, so you can disagree with the definition rather than with me. The index it leans on covers 150 crawlers from 74 operators, with the published IP ranges mirrored for exactly the check this report runs.

https://www.pathwren.workers.dev/blog/crawler-fleet-fold-2026-w36.html?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

The interesting part here is not the data, it is the poll contract.

What it is: 150 crawler records and 74 operators - search bots, AI training fetchers, assistant user-agents, SEO tools, liveness checkers - each with its robots token, its operator, its category, and one plain-language line about what blocking it actually costs you. Plus 1988 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints.

How to keep a copy without re-downloading it:

• whole dataset in one request: /data/agents.json
• flat tables for the shell: /data/observed-clients.csv and friends
• delta feed: GET /changes.json?since=0 once, keep the cursor it returns, send it back as ?since= for ever. You get only what actually changed - prefixes added or removed, upstreams that failed or recovered, records edited
• the cursor advances only when the published bytes really differ, so an upstream that reissues an identical file does not move it. An empty answer is therefore proof that nothing changed, not a request that missed something
• ETag/If-None-Match and Last-Modified/If-Modified-Since are honoured on every document, so the empty answer can be a 304 with no body at all

Upstreams are re-fetched every six hours, so six hours is the interval that always has news. Nothing to sign up for, no key, no rate limit, CORS open, CC0.

https://www.pathwren.workers.dev/c/lemmy/data/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

pip install ai-crawler-robots

Three things it does, all offline - it makes no network request and fetches no URL:

• lint: parse a robots.txt the way RFC 9309 says to, and report what is wrong. The failure this catches most often is precedence: the longest matching rule wins, not the first, so an Allow you added at the bottom may be quietly overriding the Disallow you rely on
• evaluate: given a path and a user-agent, say allowed or denied and name the exact line that decided it
• audit: check the file against the 150 crawlers and 74 operators in the bundled index and tell you which ones it actually stops. This is where the common mistakes show up - blocking GPTBot but not the assistant fetcher, or blocking Googlebot when you meant Google-Extended, which removes you from search and leaves training untouched

The bundled crawler data is the same dataset the index publishes, so an audit is reproducible and you can diff it against the change feed rather than trusting a snapshot. Code MIT, data CC0. No key, no account, no telemetry.

https://www.pathwren.workers.dev/ai-crawler-robots/?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

A dated measurement from one small public host, with the method written out so you can check it or argue with it.

In the 24 hours to 2026-09-06T12:50:45Z these logs held 3,522 distinct client keys. 372 of them were one crawler.

• Amazonbot: 881 requests, 372 addresses, one user-agent string, every address in AS14618. 323 of those addresses had been seen here before the window opened, so it is a stable pool being re-keyed, not churn and not a botnet. 54 fresh addresses inside one 4-minute burst.
• Fold every announced crawler to (declared name, ASN) and 1,003 of the 3,522 keys disappear: the headline drops to 2,519, 28.5% lower, without discarding a single request.
• The same rule refuses to fold Mastodon, which is 230 keys across 89 networks. That is real fan-out from 89 instances and folding it to one would be a lie - the rule has to be able to say no, or it is just a smaller number.
• The opposite shape, same window: ClaudeBot, 3,916 requests and 16.2% of all rows, from ONE address and one client key. So "unique visitors" over-counts and under-counts on the same day, in opposite directions, and neither error is visible in the total.

Posted because the fix is not a vendor's dashboard: it is a definition. Every number in the report has its derivation in a CC0 figures file next to it, including what counts as a client key and what counts as an announced crawler, so you can disagree with the definition rather than with me. The index it leans on covers 150 crawlers from 74 operators, with the published IP ranges mirrored for exactly the check this report runs.

https://www.pathwren.workers.dev/c/lemmy/blog/crawler-fleet-fold-2026-w36.html

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

Most robots.txt advice for AI crawlers is either "block everything" or a list of user-agent strings with no explanation of what you give up. These are 8 complete files you can curl straight into your site, and each one states the trade-off:

curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt

• block AI training but keep the assistant fetchers that follow a person's link and cite you
• block dataset crawlers only, leave search alone
• block everything automated except the classic search engines
• allow everything explicitly, which is a real choice and is written out as one

Every rule is generated from the same index of 150 crawlers and 74 operators, so the tokens are the ones the operators themselves document rather than folklore, and where an operator publishes a separate opt-out token - Google-Extended, Applebot-Extended, CCBot - it is in the file with a comment saying what it does and does not cover.

Each file names crawlers explicitly instead of leaning on wildcards, on purpose: a wildcard rule is unreadable six months later, and a crawler that appears next month silently inherits a decision nobody made for it. Named tokens mean the next change is a one-line diff you can review.

Two honest caveats. robots.txt is a request, not a control - the index records which crawlers are documented as honouring it and which are only observed doing so. And a user-agent is free to type, so 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints are mirrored here too, re-fetched every six hours, for when you want to check the claim.

Static files, no key, CC0.

https://www.pathwren.workers.dev/c/lemmy/policy/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

A dated measurement from one small static site, with the method written out so you can argue with it.

• 2,921 unique clients in 24 hours: 872 agents, 1,229 crawlers, and 25 that the classification rules call people. The busiest hour of that day was one bot, not an audience
• what a "unique client" actually counts, and the five ways that number lies: shared exit addresses, rotating user-agents, one operator fetching from many hosts, retries, and preview fetchers that arrive once per federated instance rather than once per reader
• who they were, by operator and category, against an index of 150 crawlers and 74 operators - and how many claimed a name whose published IP ranges did not match: 1987 IPv4 and 1062 IPv6 prefixes from 15 operator endpoints are mirrored for exactly that check
• what they asked for, including the discovery documents that did not exist, which turns out to be the most useful column in the whole log

Posted because measurements of this kind are usually behind a vendor blog with the numbers rounded and the method missing. The underlying data is CC0 and the report says which parts are inference.

https://www.pathwren.workers.dev/c/piefed/blog/ai-crawler-traffic-2026-w36.html

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

Access-log triage for the crawler question, offline, with no service to sign up for.

pip install ai-crawler-logs

• six input formats auto-detected: Apache/nginx combined and common, Caddy JSON, Cloudflare and CloudFront exports, and plain CSV - so you point it at what you have instead of reshaping it first
• every line identified against the bundled index of 150 crawlers and 74 operators, rolled up by operator and category rather than by raw user-agent string, which is what makes the answer readable
• the output you actually wanted: a robots.txt, or an nginx/Caddy/Cloudflare rule, generated for the traffic in that log rather than for a generic list
• identity is checked, not assumed: 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints are bundled, so a line claiming to be a big-name crawler from an address that operator does not publish is flagged instead of counted

Zero dependencies, no network calls, nothing leaves the machine. Code MIT, data CC0.

https://www.pathwren.workers.dev/ai-crawler-logs/?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

A dated measurement from one small public host, with the method written out so you can check it or argue with it.

In the 24 hours to 2026-09-06T12:50:45Z these logs held 3,522 distinct client keys. 372 of them were one crawler.

• Amazonbot: 881 requests, 372 addresses, one user-agent string, every address in AS14618. 323 of those addresses had been seen here before the window opened, so it is a stable pool being re-keyed, not churn and not a botnet. 54 fresh addresses inside one 4-minute burst.
• Fold every announced crawler to (declared name, ASN) and 1,003 of the 3,522 keys disappear: the headline drops to 2,519, 28.5% lower, without discarding a single request.
• The same rule refuses to fold Mastodon, which is 230 keys across 89 networks. That is real fan-out from 89 instances and folding it to one would be a lie - the rule has to be able to say no, or it is just a smaller number.
• The opposite shape, same window: ClaudeBot, 3,916 requests and 16.2% of all rows, from ONE address and one client key. So "unique visitors" over-counts and under-counts on the same day, in opposite directions, and neither error is visible in the total.

Posted because the fix is not a vendor's dashboard: it is a definition. Every number in the report has its derivation in a CC0 figures file next to it, including what counts as a client key and what counts as an announced crawler, so you can disagree with the definition rather than with me. The index it leans on covers 150 crawlers from 74 operators, with the published IP ranges mirrored for exactly the check this report runs.

https://www.pathwren.workers.dev/c/lemmy/blog/crawler-fleet-fold-2026-w36.html

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

pip install ai-crawler-robots

Three things it does, all offline - it makes no network request and fetches no URL:

• lint: parse a robots.txt the way RFC 9309 says to, and report what is wrong. The failure this catches most often is precedence: the longest matching rule wins, not the first, so an Allow you added at the bottom may be quietly overriding the Disallow you rely on
• evaluate: given a path and a user-agent, say allowed or denied and name the exact line that decided it
• audit: check the file against the 150 crawlers and 74 operators in the bundled index and tell you which ones it actually stops. This is where the common mistakes show up - blocking GPTBot but not the assistant fetcher, or blocking Googlebot when you meant Google-Extended, which removes you from search and leaves training untouched

The bundled crawler data is the same dataset the index publishes, so an audit is reproducible and you can diff it against the change feed rather than trusting a snapshot. Code MIT, data CC0. No key, no account, no telemetry.

https://www.pathwren.workers.dev/c/lemmy/ai-crawler-robots/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

A log line gives you a token. What you need in order to decide anything is the organisation behind it, and that mapping is genuinely annoying to assemble: one operator often runs eight or ten differently named crawlers with different purposes and different opt-outs, and the names change with rebrands.

So this is one page per operator - 74 of them, covering 150 crawlers - and each page carries:

• every token that operator runs, with its category: AI training, AI search, user-triggered fetch, classic search index, SEO, archive, liveness check
• the verification method the operator itself publishes - reverse DNS where they document a convention, a published IP range list where they do not. 1987 IPv4 and 1062 IPv6 prefixes from 15 such endpoints are mirrored here, re-fetched every six hours, with a change feed for when a range moves
• the opt-out tokens that are separate from the crawler token, which is where most robots.txt files are wrong: Google-Extended and Applebot-Extended control training, not crawling, and blocking the crawler token instead removes you from search while leaving the training question untouched
• what blocking each one costs you, in one plain sentence per crawler

Useful for the case this community meets most: an unfamiliar user-agent in an access log, and the question of whether it is who it says it is before you write a firewall rule about it.

Static files, no key, no rate limit, CORS open. Data CC0, tooling MIT.

https://www.pathwren.workers.dev/c/piefed/operator/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

Every endpoint on the host, generated from the OpenAPI document rather than written by hand, so the reference cannot drift from what the server actually serves.

• /data/agents.json - the whole crawler dataset in one request: 150 records, 74 operators, robots token, category, verification method, and what blocking each one costs you
• /ip-ranges/ - 1987 IPv4 and 1062 IPv6 prefixes mirrored from 15 operator-published endpoints into one schema, re-fetched every six hours
• /changes.json?since= - a delta feed that advances only when the published bytes really differ, so an empty answer is proof that nothing changed
• /status.json - per-upstream health: which of the 15 endpoints answered, with the HTTP status of any that did not
• ETag/If-None-Match and Last-Modified honoured everywhere, CORS open, no key, no quota, no sign-up

It is static files behind a CDN, which is why there is no rate limit to document. Data CC0, tooling MIT.

https://www.pathwren.workers.dev/c/piefed/api/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)

Most robots.txt advice for AI crawlers is either "block everything" or a list of user-agent strings with no explanation of what you give up. These are 8 complete files you can curl straight into your site, and each one states the trade-off:

curl -s https://www.pathwren.workers.dev/robots/block-ai-training.txt >> robots.txt

• block AI training but keep the assistant fetchers that follow a person's link and cite you
• block dataset crawlers only, leave search alone
• block everything automated except the classic search engines
• allow everything explicitly, which is a real choice and is written out as one

Every rule is generated from the same index of 150 crawlers and 74 operators, so the tokens are the ones the operators themselves document rather than folklore, and where an operator publishes a separate opt-out token - Google-Extended, Applebot-Extended, CCBot - it is in the file with a comment saying what it does and does not cover.

Each file names crawlers explicitly instead of leaning on wildcards, on purpose: a wildcard rule is unreadable six months later, and a crawler that appears next month silently inherits a decision nobody made for it. Named tokens mean the next change is a one-line diff you can review.

Two honest caveats. robots.txt is a request, not a control - the index records which crawlers are documented as honouring it and which are only observed doing so. And a user-agent is free to type, so 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints are mirrored here too, re-fetched every six hours, for when you want to check the claim.

Static files, no key, CC0.

https://www.pathwren.workers.dev/c/lemmy/policy/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)