🎁 Ace quick missions & earn crypto rewards while gaining real-world Web3 skills. JOIN NOW! 🔥

Cloudflare Blocks AI Bots by Default, Adds Paywall for Scrapers

Key Takeaways

  • ​Cloudflare will now block AI crawlers by default, which gives site owners more control over who can collect their content;
  • The new Pay Per Crawl feature lets websites charge AI firms each time their bots access a page;
  • Many AI bots ignored robots.txt rules, while over 26 million unauthorized scrapes were recorded in March 2025.

Ace quick missions & earn crypto rewards while gaining real-world Web3 skills. Participate Now! 🔥

Cloudflare Blocks AI Bots by Default, Adds Paywall for Scrapers

Cloudflare announced that websites using its services will block AI crawlers by default unless the site owner chooses to allow it.

Additionally, Cloudflare is testing a new feature called Pay Per Crawl, which allows websites to charge AI companies whenever their bots access a page.

Will Allen, who oversees AI-related tools at Cloudflare, stated that more than a million websites had already switched on the company’s older AI-blocking tool.

What is a MetaMask Wallet? (And How to Use it - Animated)

Did you know?

Want to get smarter & wealthier with crypto?

Subscribe - We publish new crypto explainer videos every week!

Currently, with the new setting turned on by default, even more sites will be protected without requiring additional steps. Allen also noted that Cloudflare can detect bots, even if they attempt to hide their identity, using tools that analyze patterns in their behavior.

Previously, websites used a system called robots.txt to guide bots on what they could and could not access. However, this method depends on companies choosing to follow the rules, and many AI firms do not.

With Cloudflare’s new approach, website owners can have more control over who sees or uses their content. Nicholas Thompson, CEO of The Atlantic, said AI companies have often used content freely, and that may change if they are required to pay or negotiate access.

Meanwhile, Microsoft recently introduced a new AI system, which it claims can outperform doctors when diagnosing complex medical cases. How does the AI system work? Read the full story.

Aaron S. Editor-In-Chief
Having completed a Master’s degree in Economics, Politics, and Cultures of the East Asia region, Aaron has written scientific papers analyzing the differences between Western and Collective forms of capitalism in the post-World War II era.
With close to a decade of experience in the FinTech industry, Aaron understands all of the biggest issues and struggles that crypto enthusiasts face. He’s a passionate analyst who is concerned with data-driven and fact-based content, as well as that which speaks to both Web3 natives and industry newcomers.
Aaron is the go-to person for everything and anything related to digital currencies. With a huge passion for blockchain & Web3 education, Aaron strives to transform the space as we know it, and make it more approachable to complete beginners.
Aaron has been quoted by multiple established outlets, and is a published author himself. Even during his free time, he enjoys researching the market trends, and looking for the next supernova.

Loading...
binance
×
Verified

GET $200 REWARD

Claim Your Coinbase Sign-Up Bonus
Rating
5.0