This is a column about technology. See my full ethics disclosure here.
Cloudflare’s Matthew Prince says Perplexity is crawling like a thief in the night. Perplexity’s CEO, Aravind Srinivas, says: not so fast.
According to Perplexity, their AI isn’t a traditional crawler vacuuming up the web. It’s more like a proxy acting on behalf of a user. When you ask a question, the system fetches the answer in real time, just as you could by visiting a site yourself. If a human can access it, their agent should too.
Perplexity’s Case
Agents aren’t bots. They extend user intent. Blocking them is like blocking the user.
Third-party misfires. Some flagged activity may have come from services like BrowserBase, not Perplexity directly.
Cloudflare’s theatrics. Perplexity spokesperson Jesse Dwyer dismisses the “stealth crawler” charges as exaggerated, even a publicity stunt.
The Core Divide
At the heart of this is a deceptively simple question:
If an AI acts on behalf of a human, does it inherit the human’s right to access content—or does it need its own license to exist?
Cloudflare says enforcement is the only way to protect the economics of content. Perplexity says freer access is essential, or innovation dies.
Why This Matters
There’s a clear line in the sand here. Cloudflare believes AI should license content according to the creator’s wishes, whereas Perplexity believes that agentic behavior should inherit the same rights as the human directing the activity.
In plain terms, Cloudflare thinks that creators should have ownership, whereas Perplexity thinks agentic tools should have free rein.
The takeaway for businesses: your site’s visibility may soon hinge less on rankings and more on whether AI agents can get through the gate.
Perplexity is making the case that bypassing the creators’ directives is acceptable because they are acting on behalf of users, but it’s not hard to see how this is a slippery slope. What we’re seeing play out is the free market responding to creators wanting tools to combat what they view as delinquent bots and crawlers.
“This controversy reveals that Cloudflare's systems are fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats. If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic” (Perplexity).
The Big Question
Would you rather see a licensed model that risks starving small creators — or an open field where AI companies run free?
This is Part 2 of the story. If you missed Part 1, I unpacked Cloudflare’s case for blocking AI crawlers and why they think the open web’s economics are at risk.
Next: Part 3 - What This Fight Tells Us About the Future of AI and the Web