Cloudflare Launches ‘Pay per Crawl’ to Monetize AI Access to Web Content

Cloudflare has unveiled Pay per Crawl, a new marketplace designed to let publishers control and monetize how AI companies access their content. The initiative, now in private beta, allows website owners to set pricing for AI crawlers on a per-crawl basis, block them entirely, or grant access for free. The platform aims to introduce a sustainable, publisher-centric model as AI systems increasingly source data directly from websites.

The launch follows a year of AI-focused tooling by Cloudflare, including one-click crawler blocks and visibility dashboards. Backed by major publishers such as TIME, The Atlantic, and Conde Nast, the marketplace positions Cloudflare as a central broker in a shifting digital economy, where AI agents replace traditional search-driven traffic.