This is a column about technology. See my full ethics disclosure here.
For decades, the web ran on a quiet handshake: publishers made content, search engines indexed it, and traffic flowed back in return. It wasn’t perfect, but it kept the lights on.
That handshake is breaking as AI strip mines the internet
Cloudflare CEO Matthew Prince recently accused Perplexity, one of the rising AI “answer engines,” of violating those norms. Instead of politely respecting robots.txt and declared bot identities, Cloudflare found Perplexity’s crawlers sneaking in through the side door. They rotated IPs, disguised themselves as Chrome browsers, and ignored explicit “no-crawl” directives. Prince compared the tactics to “North Korean hackers”.
Cloudflare’s Case
Robots.txt and AI.txt are the baseline. Disobeying it isn’t innovation; it’s theft dressed up as progress.
Stealth crawlers remove any ounce of trust. Bots impersonating humans erode the very system of open access that the web depends on.
Economics matter. Search engines once sent traffic back to publishers. AI engines keep it, synthesizing content without referral clicks. Cloudflare’s counter: block AI bots by default and create a “Pay-Per-Crawl” system where publishers set terms.
Cloudflare even ran sting operations: planting hidden, no-crawl domains. When those pages still surfaced in Perplexity’s answers, the case got stronger.
Why Prince is Pushing This
The open web isn’t free — it’s subsidized by traffic. Remove the traffic, and you remove the funding model for journalism, research, and most of what’s worth reading online.
Prince frames it bluntly: either we pay creators fairly for the data that fuels AI, or we hollow out the web until nothing valuable is left.
To put it plainly: if AI agents are going to sneak past the velvet rope, they’d better start tipping the bartender.
The Big Question
This fight isn’t just Cloudflare versus Perplexity. It’s about who sets the rules for the next web era: infrastructure providers, AI companies, or regulators scrambling to catch up.
This is only part 1 of the story.
Next in part 2 - Perplexity’s rebuttal, and why they think Cloudflare is overreacting.