Cloudflare's new policy pushes AI companies to pay for publishers' content

That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says. The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services.

Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protections against having their intellectual property given away for free. Cloudflare specifically calls out the "world's largest search engine" (clearly a Google reference!) as having access to about "2x more information" than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI.

Google has pushed back against this generalization in the past, noting that it provides a bot called Google Extended that lets site owners opt out of having their content used for training and AI products and services like Gemini Apps and Vertex API. Its use doesn't impact a site's inclusion in Google Search. However, the tech giant's flagship Googlebot crawls for Search, including AI features like AI Overviews and AI Mode.