Cloudflare to block AI training crawlers on ad supported pages

Cloudflare will begin blocking mixed use crawlers, bots that collect data for both search indexing and AI training, from ad supported customer pages starting September 15, 2026. The shift moves web crawling from an implicit permission model to an explicit one, raising the operational cost for teams that rely on large scale web scraping for training data. Cloudflare, which manages around 20 percent of global web traffic, is also piloting a pay per crawl system allowing publishers to charge AI companies for access. The company argues that the traditional crawl for traffic exchange has broken down, citing extreme crawl to referral ratios for AI companies compared to search engines. For machine learning practitioners, the change means pipelines must incorporate opt in checks, credentials, or paid access, making data sourcing more complex and expensive. Industry reactions show strong publisher support, and the move may accelerate the shift toward licensed content marketplaces.

Read the full story on Engadget →