What This Fight Tells Us About the Future of AI and the Web
The Cloudflare/Perplexity dispute, Part III
This is a column about technology. See my full ethics disclosure here.
The Cloudflare–Perplexity clash has been framed as a technical fight about crawlers, agents, and robots.txt. But zoom out, and the debate reveals something deeper: how we want the AI internet to work.
Cultural framing
Perplexity is recasting itself as a consumer tool, not a crawler. That’s a clever shift. It positions them alongside users, i.e., “we’re just acting on your behalf,” and casts Cloudflare as the heavy-handed gatekeeper. This reframing may resonate culturally, but it dodges the core question: Is an AI really just you, extended, or a separate entity that needs its own rules?
Legal and political implications
If agents inherit human-like rights, companies can justify massive scraping operations under the banner of user intent. That creates loopholes large enough to drive a server farm through. On the flip side, if agents are treated as distinct actors, new licensing models — or outright restrictions — are inevitable. Either way, the law will need to catch up, and the outcome will ripple into copyright, antitrust, and consumer rights.
Industry risks
The economics aren’t abstract.
Open roaming strips content creators and publishers of content without compensation.
Licensing favors giants who can afford to cut deals, but the middle of the market, the independent publishers, niche media, and smaller platforms are at risk of being crushed whichever way this breaks.
Cloudflare’s angle
It’s worth noting Cloudflare’s report wasn’t about “agentic AI” in general. The issue is Perplexity’s alleged workarounds: stealth crawlers and obfuscated tactics. Cloudflare claims that when it tested OpenAI’s ChatGPT Agent under the same conditions, it observed no attempts to bypass blocks. That distinction matters. It suggests this isn’t about whether agents can respect boundaries — but whether specific companies will.
Why this debate matters
This divide cuts to the heart of how the AI internet functions. Notably, Perplexity didn’t deny bypassing robots.txt, rather it argued the behavior was justified: if you wouldn’t deny the content to a person, you shouldn’t deny it to a bot acting for that person.
That’s the fault line. Do we treat agents as humans with extended reach, or as distinct actors with their own obligations? The answer will define how information retrieval, the media industry, and digital rights evolve.
The fight between Perplexity and Cloudflare boils down to a simple but loaded question: who decides access? Do websites get the right to block agents, or can a person send an AI on their behalf and expect it to walk through the same door they can?
The answer matters. If agents inherit user rights, as Perplexity argues, the web tilts quickly toward a bot-first internet, where actual human visits decline. And if that happens, site owners won’t sit back. You can expect more hard paywalls and stricter monetization to protect what’s left of their traffic.
To put it bluntly: if AI agents are just us with longer arms, we need to decide how far those arms can reach before they’re picking pockets.
The Big Question
Do we want an AI internet governed by “human proxy” rights or a system where agents must play by separate, enforceable rules?
This was the last piece for now. We’ll have to see how it shakes out.
Part 1 - When AI Breaks the Rules