47
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access (blog.cloudflare.com)
submitted 9 months ago by to c/webdev@programming.dev
7 comments fedilink hide all child comments
 

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:

  1. Allow: Grant the crawler free access to content.
  2. Charge: Require payment at the configured, domain-wide price.
  3. Block: Deny access entirely, with no option to pay.

Although this is old news, I still found it interesting. Also, I like the "AI Labyrinth" feature of Cloudflare to block AI bots.

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 9 months ago

Cloudflare's business model is dealing with unethical and unlawful bots. They can pour money into the arms race with "AI" companies and win. If they can't their service isn't valuable.

  • permalink
  • fedilink
  • source
  • parent