Cloudflare has announced that from September 15, 2026, its default settings will block mixed-use web crawlers from pages that carry advertising, drawing a hard line between bots used for traditional search indexing and those used for AI agent operations and model training.
The change will apply automatically to new customers, new sites created by existing customers, and all existing free-tier users. Site owners will retain the ability to adjust settings manually, but the default posture shifts control firmly toward publishers. Co-founder and CEO Matthew Prince framed the move as a necessary response to bot traffic surpassing human traffic online for the first time, a milestone not expected until next year.
Cloudflare singled out Google for criticism, arguing that the search giant’s dominance gives it access to roughly twice the web content available to other AI companies, because remaining discoverable in search effectively requires consenting to AI use. Google has previously pointed to its Google Extended opt-out tool as a rebuttal to such claims.
Alongside the crawler policy, Cloudflare announced the evolution of its Pay Per Crawl marketplace into a Pay Per Usemodel, which would allow publishers to collect revenue when their content generates value for AI systems, not merely when it is fetched. Initial partners include Ceramic.ai and You.com. The company also noted that more than half of AI crawler traffic is currently spent re-fetching pages that have not changed, representing significant wasted bandwidth for publishers.
