Cloudflare to Blocks AI Bots by Default

Crypto News AI

Cloudflare Blocks AI Bots by Default, Adds Paywall for Scrapers

Written by Aaron S.,

Editor-In-Chief

Last Updated: July 02, 2025

Key Takeaways

Cloudflare will now block AI crawlers by default, which gives site owners more control over who can collect their content;
The new Pay Per Crawl feature lets websites charge AI firms each time their bots access a page;
Many AI bots ignored robots.txt rules, while over 26 million unauthorized scrapes were recorded in March 2025.

Stop overpaying - start transferring money with Ogvio. Join the waitlist & grab early Rewards NOW! 🎁

Cloudflare Blocks AI Bots by Default, Adds Paywall for Scrapers

BITDEGREE IN YOUR SOCIAL FEED

Cloudflare announced that websites using its services will block AI crawlers by default unless the site owner chooses to allow it.

Additionally, Cloudflare is testing a new feature called Pay Per Crawl, which allows websites to charge AI companies whenever their bots access a page.

Will Allen, who oversees AI-related tools at Cloudflare, stated that more than a million websites had already switched on the company’s older AI-blocking tool.

How to Pick the Right DeFi dApp? (Dos and Don’ts Explained)

Did you know?

Want to get smarter & wealthier with crypto?

Subscribe - We publish new crypto explainer videos every week!

Currently, with the new setting turned on by default, even more sites will be protected without requiring additional steps. Allen also noted that Cloudflare can detect bots, even if they attempt to hide their identity, using tools that analyze patterns in their behavior.

Previously, websites used a system called robots.txt to guide bots on what they could and could not access. However, this method depends on companies choosing to follow the rules, and many AI firms do not.

With Cloudflare’s new approach, website owners can have more control over who sees or uses their content. Nicholas Thompson, CEO of The Atlantic, said AI companies have often used content freely, and that may change if they are required to pay or negotiate access.

Meanwhile, Microsoft recently introduced a new AI system, which it claims can outperform doctors when diagnosing complex medical cases. How does the AI system work? Read the full story.