One of our core values at NerdPress is to Teach & Empower. I’m thrilled to be writing this post because what we’ve just launched is truly empowering for our clients.
The explosion of AI in recent years has created a huge challenge for content creators. Bots are crawling websites at unprecedented rates—training on massive amounts of original, human-created content and then regurgitating it in “AI Overviews” or other chat-like interfaces.
(Note: No AI was used to write this blog post. Every emdash is my own!)
Generative AI may be considered useful, but at its core, it’s extremely sophisticated plagiarism. Even worse, it’s taking traffic away from the original content creators, impacting their ability to make a living.
We believe that creators should be compensated fairly for their work.
If original content creators go away, the AI bots will have nothing to “feed” off of, and LLMs will just become giant echo chambers (even more than they are now). So the AI companies know they are going to need to pay; licensing deals between Amazon and the New York Times and Anthropic’s $1.5B settlement with book publishers are two high-profile examples.
But what about smaller, independent content creators? Without huge legal departments, we need other solutions.
Some are saying that it’s too late—that the models have already been trained on your data. But 76% of AI Bot traffic right now is still for training crawls.
Or maybe it’s better to “just embrace” AI and hope to get clicks from the AI Overviews and chatbots. If that’s working well for you, then great! But from what we’ve seen so far, the AI platforms are not sending enough traffic back to sites to make it worthwhile.
So, should you just block the bots? Even if you want to do that, it’s much harder than it seems.
Traditional Methods Aren’t Enough
Historically, reputable crawlers would follow directives in the robots.txt file. But that’s just a suggestion, really, and some AI companies respect those instructions while others don’t.
Many solutions block AI traffic based only on User Agent strings—but it’s trivial to spoof User Agents.
What to do?
Enter Enterprise Bot Management
Because ~20% of the internet flows through Cloudflare’s servers, they’re able to detect and identify bots based on crawl behavior and other signals, not just User Agent or IP address.
With AI Crawl Control and Enterprise Bot Management, Cloudflare assigns each bot a Bot Detection ID. We then create firewall rules with those IDs—blocking them at the network level with extremely high accuracy.
“Until now, Cloudflare’s advanced bot management and blocking features were hard to manage for most independent publishers. By making enterprise-grade protection accessible, NerdPress is empowering creators to protect our work and take a united stand against unauthorized AI scraping.”
It Starts with Blocking
Today, I’m excited to announce that we can now leverage Enterprise Bot Management to block AI crawlers for any of our clients on a per-hostname and per-crawler basis.
(This will be opt-in because we want every site owner to decide what’s best for them.)
Want to block Perplexity, but allow AppleBot? No problem.
Want to block all AI Training, but not AI Assistants or AI Agents? We can do that, too.
Or if you’re not sure what to block, you can opt in to one of our managed block lists, and we’ll just take care of it. We’ve developed our custom list in consultation with Raptive and others and will continue to tweak what we’re blocking as things evolve.
“Raptive has long advocated for the rights of independent creators, and this collaboration with NerdPress marks a meaningful move in the fight for a fairer internet. AI companies are building trillion-dollar technologies on the backs of creators, and it’s past time they start paying for the content that powers their systems. Blocking unauthorized scraping is a critical step toward restoring control — and value — to the people who make the open web worth visiting.”
Like I said: Empowering!
The Road to Pay Per Crawl
Blocking, when done correctly, is the first step towards creators getting paid for their work. It sends a clear signal to the crawlers that they’re going to need to pay to access the content. Today, there is no unified system for this—but methods are being actively developed and tested by Cloudflare and others. (We’re participating in other “proof of concept” betas as well.)
For now, when we issue a block, we don’t send a traditional 403 Forbidden response. Instead, we’re sending a 402 Payment Required response, with a simple message:
Free crawling is prohibited. Please contact the site owner for licensing options.
And, as Pay Per Crawl (or similar) tools become available—and hopefully those new standards will emerge soon—you can bet we’ll be hard at work to make them readily available to our clients, too.
Available Now!
AI Bot Blocking is available now to all our clients who are currently using our Cloudflare Enterprise service. If you’re an existing client, just email us to let us know which bots you’d like to block.
We’re also working on a settings page which will be available right in the WordPress dashboard, to give you direct control over blocking, along with insight from bot analytics.
(One note: We’re calling this a “beta” for the moment and are offering it for no extra charge, at least while it’s in beta.)
Since the landscape is evolving rapidly, we’ll keep improving as the industry changes—because we’re with you, every step of the way.
–
Not yet a NerdPress client? We’d love to empower you, too, of course! You can learn about our WordPress Support plans, and we’d be happy to answer any questions.
