Cloudflare’s New Marketplace Will Let Websites Charge AI Bots

Cloudflare, a leading provider of internet infrastructure and security services, has announced a groundbreaking initiative to empower website owners to control and monetize how their content is used by AI models. The company plans to launch a marketplace within the next year, allowing websites to charge AI bots for scraping their content, a move aimed at addressing the growing concerns about the impact of AI on online content creation.

At the heart of this initiative is Cloudflare's new tool, AI Audit, which is available immediately to all Cloudflare customers. AI Audit provides website owners with detailed analytics on AI bot activity, including information on which companies are scraping their content, how frequently, and which sections of their sites are most popular among AI crawlers. This tool categorizes AI-related bots into three types: AI Data Scrapers, AI Search Crawlers, and Archivers, each of which can impact a website differently. For instance, AI search engines might drive traffic to a site, while data scrapers primarily collect information for model training without necessarily attributing the source.

Cloudflare has also introduced a one-click option to block all AI crawlers, giving site owners a "pause" to assess their strategy regarding AI content usage. For more granular control, administrators can create custom rules to allow or block specific AI providers or bot types. This feature addresses the previous limitation where website owners could only block all AI bots with a single button, without the ability to differentiate between various crawlers.

The upcoming marketplace will enable content creators to set prices for AI companies to access and scrape their sites. This move is part of a broader effort to give publishers more control over how AI bots interact with their websites. Cloudflare CEO Matthew Prince emphasized the need for fair compensation, stating, "We think that sites of any size should be fairly compensated for the use of their content." This approach is seen as crucial for the long-term sustainability of the AI ecosystem and the broader internet.

Currently, only large publishers have been able to negotiate deals with major AI companies like OpenAI. Cloudflare's platform could extend this opportunity to smaller content creators, handling the pricing, access control, and value capture for content scanning. This democratization of content licensing could be a significant shift, allowing smaller websites to benefit financially from the use of their content by AI models.

The new tools also help website owners who have already negotiated deals with AI companies. Cloudflare provides the ability to generate reports to verify if AI providers are adhering to agreed-upon terms regarding scanning frequency and content access. This ensures transparency and accountability in the licensing agreements between website owners and AI model providers.

Cloudflare's unique position, with its network underlying a significant portion of the global internet, makes it an ideal intermediary between content creators and AI model providers. Prince highlighted that Cloudflare's relationships with major AI companies create a two-sided market, where the company can set rules that AI companies must adhere to, given its significant impact on the web.

The need for such a marketplace is underscored by the increasing complaints from website owners about excessive AI scraping. Some have reported server issues resembling DDoS attacks due to the high frequency of AI bot activity. For example, repair database iFixIt complained that a web crawler bot for Anthropic’s AI chatbot Claude hit its website nearly a million times in a single day.

Cloudflare customers can begin using the AI Audit tools immediately through their dashboards and are invited to join the waitlist for the marketplace’s beta version. The company will further showcase these advancements during its Builder Day Live Stream on September 26.

In summary, Cloudflare's new marketplace and tools are designed to give website owners granular control over AI bot activity, provide detailed analytics, and offer a platform for monetizing their content. This initiative aims to address the imbalance in the AI ecosystem where many websites are scraped without compensation or even awareness, ensuring a more sustainable future for content creation online.