Cloudflare announced on Wednesday that starting September 15, 2026, its default settings will block “mixed-use” crawlers – bots that blend search indexing, AI agent use, and training data collection in a single process – from any pages that host advertising on Cloudflare-protected sites. The change applies to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers. Site owners can adjust the default, but the baseline position will shift from permissive to restrictive for AI-and-agent crawling on ad-supported pages. Simultaneously, Cloudflare announced the evolution of its existing Pay Per Crawl programme, launched exactly one year earlier on July 1, 2025, into a broader Pay Per Use model under which publishers can receive payment when their content contributes to an AI-generated answer rather than simply when a crawler fetches their page. Ceramic.ai and You.com are the first partners in the Pay Per Use programme. YourNewsClub finds the September 15 default-block date more commercially consequential than the Pay Per Use announcement: a platform default that affects millions of sites simultaneously creates legal and operational urgency for AI companies that a voluntary monetisation programme does not.
Cloudflare protects approximately 20% of the global web, and its announcement explicitly says publishers “should not have to choose between being discoverable online and giving away their work for free to AI systems.” When a default changes at that infrastructure layer, the aggregate effect on AI crawler access is meaningful even before any individual publisher adjusts their own settings.
AI companies have not consistently disclosed whether a given crawler collects training data, serves agent requests, or indexes for search – many serve all three from the same IP range and user-agent string. Cloudflare is introducing finer-grained classification tools giving site owners separate controls for search, agent, and training bots. YourNewsClub pins that classification granularity as the most practically useful element for publishers: a site can remain discoverable in AI search results while blocking the same crawler from scraping content for model training.
Owen Radner, who models digital infrastructure as energy-information transport systems, draws the infrastructure layer argument: “Cloudflare is setting defaults at the network layer that effectively create friction costs for AI companies that do not separate their crawler purposes. The September 15 date gives those companies ten weeks to build separate search, agent, and training crawlers or accept that mixed-use bots will be blocked from a significant fraction of ad-supported web content.” Alex Reinhardt, who tracks financial systems and settlement infrastructure through digital protocols, places the payment model transition: “Pay Per Use is economically rational for publishers but operationally dependent on attribution. AI answer engines that do not cite sources cannot participate, which means the programme will initially reward companies that already practice transparent citation over those that produce unattributed answers.”
Cloudflare also announced a planned Attribution Business Insights dashboard that will show how AI bots access content, where that content is cited, and how much human traffic different AI platforms return to publisher sites. That analytics layer – showing publishers the actual downstream traffic value of AI citation – does not yet exist for most publishers. YourNewsClub rates that dashboard, when it ships, as the tool most likely to shift publisher attitudes from passive resistance to AI crawling toward active negotiation with specific AI platforms on specific terms.
Cloudflare’s position is unusual: it sits between publishers and AI companies as network infrastructure, not as a participant on either side. Its ability to set default access rules at infrastructure level – without being party to any content agreement – is the structural power that makes the September 15 announcement more consequential than a voluntary industry standard.
The programme is described as neutral by design and restricted to search functions: no publisher content is shared through the Pay Per Use programme itself, and none of the data is used to train foundation models. Full broad availability is planned for later in 2026 without a specific date. Your News Club signals September 15 as the moment when the announced policy becomes operational reality, since that is when site owners will first observe whether the default-block setting changes the traffic mix they receive from AI crawlers versus traditional search bots.