Cloudflare Refines AI Bot Management with Category-Based Controls

Cloudflare has revamped its AI bot management capabilities for all customers. Moving away from previous blanket blocking approaches, the company now offers a system that enables individual control by bot type. Site operators can independently toggle three categories: "Search-type," "Learning-type," and "Agent-type" bots.

This change reflects the rapid diversification of AI bots on the web. While bots have circulated online for some time, recent years have seen a surge in crawlers that collect training data for AI models and "AI agent"-style access where autonomous AI systems coordinate to retrieve information. Blanket blocking settings often result in unintended consequences, such as blocking crawlers necessary for search engine optimization.

A particularly significant change is the new default configuration taking effect from September 15, 2026. After this date, pages generating advertising revenue will automatically block "learning-type bots" and "agent-type bots." Site operators need not make individual adjustments; commercially-oriented bot traffic will be blocked by default.

This change holds important implications for the industry because Cloudflare provides infrastructure for many websites worldwide. When a company of Cloudflare's scale modifies its default settings, it means that small site operators without specialized knowledge automatically gain a level of protection. From the perspective of AI model development companies, the accessible range for training data collection may effectively narrow.

Meanwhile, the subdivision into three categories represents a departure from the binary choice of "block everything or allow everything." For instance, operators can continue allowing crawlers for search engines while restricting training data collection only—enabling flexible management. The fact that site operators can now selectively control access based on content usage intent represents a step forward from a rights protection standpoint.

Going forward, attention will focus on how AI development companies and content companies respond to the default change scheduled for September 2026. The approach to training data procurement and licensing agreements with site owners may be significantly affected. Cloudflare's decision appears poised to introduce new dimensions to ongoing discussions surrounding content use on the web.