AI Crawler Verification Barely Exists. That's Why 24 Million Fake Requests Got Through in Two Months.

Two numbers from early 2026 explain most of the AI crawler verification problem. DataDome, a bot-management vendor, logged 7.9 billion AI agent requests across January and February, a 5% rise on the previous quarter. In the same window, 16.4 million of the requests claiming to be Meta's crawler weren't. A site that thought it was letting a known AI crawler through, or charging one under Cloudflare's pay-per-crawl programme, had no reliable way to know who had actually shown up at the door.

That's the part of the crawler-economics story the pricing headlines keep skipping. Charging AI companies for access and blocking the ones you don't want both assume you can tell requesters apart. Right now, on most of the web, you can't.

The traffic nobody asked for

AI crawler traffic stopped being a rounding error some time in 2025. Training crawlers, retrieval crawlers backing chat answers, and a newer category of browsing agents that fetch a page on a user's behalf all showed up on server logs at once, and none of them were there because a human clicked a link. DataDome's count, 7.9 billion requests in two months from named AI agents alone, doesn't include the traffic that never announces itself.

For a site owner, the cost isn't abstract. Every crawl is bandwidth, compute, and origin load, and a lot of it duplicates: the same page fetched by GPTBot, ClaudeBot, PerplexityBot, and half a dozen retrieval agents within the same week, none of them coordinating with each other. Publishers noticed the bill before they noticed the philosophy question about AI training on their work, and reacted to both at once.

The category also stopped being one shape of traffic. A training crawler visits on its own schedule and never comes back for the same page twice. A retrieval crawler behind a chat answer fetches a page the moment a user asks a question about it, so its traffic looks bursty and query-shaped. A browsing agent, the kind behind products like ChatGPT Atlas or Perplexity's Comet, fetches pages live, on a person's behalf, from whatever network that person happens to be on. Treating all three as one problem, one crawler, one set of rules, is part of why the current tools cope so badly.

Two responses: block it, or charge it

The blunt option is robots.txt: list the crawler's User-Agent under a Disallow rule and hope it reads the file. Most reputable AI crawlers do. It's also, by design, unenforceable. robots.txt is a request sitting in a text file, not a lock on a door. Nothing stops a crawler from fetching the page anyway; nothing in the HTTP protocol even checks.

Cloudflare's answer, launched in 2025 and expanded through 2026, is pay-per-crawl: a site can respond to a suspected AI crawler with an HTTP 402 Payment Required status instead of the content, and set a price. An early trial against part of Stack Overflow's public dataset reportedly cut unauthorised bot traffic by around 32% and lifted data-licensing revenue by about 27%. Publishers including TIME, The Atlantic, Fortune, and Quora have signed on, and Cloudflare has said it will block AI training and agent crawlers by default on newly created domains from mid-September 2026, with site owners able to opt back in by category.

The pricing model is still moving. Cloudflare has talked about shifting from charging per fetch toward charging based on how the fetched content gets used, which is a harder thing to meter but a more honest one: a crawl that never contributes to an answer is worth less than one that does. Metering usage instead of access only raises the stakes on the identity question underneath it. If a site can't confirm which company's crawler is on the other end of a request, it can't confirm which company owes it money for reusing that content either.

Both rest on the same assumption

Blocking by name and charging by name are the same trick played in opposite directions. Both require knowing, with some confidence, which crawler sent a given request. Today that identification runs on two signals: the User-Agent header, a string the requester writes into its own request, and the source IP address, which tells you which network the packet came from and little else.

Neither is an authentication mechanism. A User-Agent string is a self-declared label, no different in kind from writing your own name on a visitor badge. An IP address identifies a network path, not an identity, and AI browsing agents increasingly route through the same consumer networks a human on their laptop uses, so the IP signal degrades exactly where it would be most useful.

robots.txt compounds the problem instead of solving it. It publishes, in plain text, exactly which User-Agent strings a site trusts or wants to keep out. For a well-behaved crawler that's a courtesy. For anyone building a scraper, it's a list of names worth impersonating.

None of this is a new flaw in a new protocol. robots.txt dates to 1994 and was never designed as access control; it was a courtesy convention among search engines that mostly wanted to be crawled, agreed among a handful of operators who all had an interest in playing along. Extending that same honour system to a much larger, much less cooperative population of AI crawlers, some of them explicitly built to extract value the source site didn't agree to give up, was always going to strain past what the convention could bear.

The numbers on faking it

DataDome's January-February 2026 data puts a figure on that incentive. Meta-ExternalAgent was the most impersonated identity in the dataset, with 16.4 million spoofed requests. ChatGPT-User was next, at 7.9 million. PerplexityBot had the highest impersonation rate of any tracked agent: close to 2.4% of requests claiming to be PerplexityBot were fraudulent.

The pattern makes sense once you separate volume from selection. A scraper doesn't pick a fake identity at random; it picks whichever name is most likely to already be sitting on a site's allowlist. Once enough sites trust a given crawler name by default, that name turns into a skeleton key, and the value of forging it scales with how many doors it opens.

What real AI crawler verification looks like

The fix Cloudflare has been building since a research post in May 2025 doesn't try to make User-Agent strings more trustworthy. It replaces them. Web Bot Auth applies RFC 9421 HTTP Message Signatures to crawler traffic: a bot operator generates a signing key pair, publishes the public half at a well-known address on their own domain, and signs the headers of every outgoing request with the private key.

The receiving site checks the signature against the published key instead of reading a self-reported string. A forged User-Agent still says whatever it wants; a forged signature simply fails to verify, because the requester doesn't hold the private key. A simplified version of what a signed request's headers look like:

GET /articles/inference-market-share HTTP/1.1
Host: example.com
Signature-Agent: "https://example-ai.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1754812800;keyid="poqkLGiymh7CDmdWOJqxrmvz24pLTztXlqQnpyzuKuc";alg="ed25519"
Signature: sig1=:g0BbBpwvKrRMoIEbJb0dSuv1RMxOO3sYNAK0Nz1qMoNaKcOMR6IhqPLzGoKEyfDx:

The site's edge (or its own verification code, since Cloudflare has open-sourced the Rust and TypeScript libraries) fetches the operator's public key from a JSON Web Key Set published under a fixed path, checks the timestamp to reject replayed requests, and confirms the signature. What it proves is narrow but real: this request was signed by whoever controls that domain's private key, full stop.

It's worth being precise about what that does and doesn't buy a site. A verified signature confirms identity, not intent. A crawler can sign every request with a legitimate key and still ignore a site's Disallow rules, scrape at a rate that hurts the origin, or reuse content in ways the publisher never agreed to. Cryptographic verification replaces a lie with a fact; it doesn't replace a policy. Sites still need rules about what a verified identity is allowed to do, the same way knowing a visitor's real name doesn't tell you whether to let them into the building.

Where sites actually stand right now

Message Signature verification is now part of Cloudflare's Verified Bots Program, rolling out first to Free and Pro tier customers with Business and Enterprise plans following. The IETF is separately weighing a dedicated working group to standardise the approach beyond one vendor's implementation, which matters: a verification scheme that only works inside Cloudflare's network protects Cloudflare's customers and nobody else.

For a site owner today, that means the honest state of the art is: large platforms and the crawlers that bother to register with them get cryptographic verification. Everyone else is still reading a header a bot chose to write, on a file the bot has no obligation to obey.

The gap for smaller sites isn't eligibility, it's attention. Cloudflare's verification libraries are open source, the key-publishing format is a standard JSON Web Key Set, and a site behind Cloudflare's edge gets signature checking without writing any code at all. What's missing for most sites under, say, a few million monthly visits is a reason to have looked into it yet, and a security or platform team with a free afternoon to configure it instead of triaging something more urgent. That's a staffing problem more than a technical one, and it's the same reason plenty of sites still run outdated TLS configurations years after the fix shipped.

The gap between the pricing story and the plumbing

The public argument this year has mostly been about money: should AI companies pay for the content they train on and cite, and how much. That's a real question, and pay-per-crawl is a genuine attempt to answer it. But every pricing scheme and every block list sits on top of an identity layer most of the web hasn't deployed. A site charging Claude's crawler per fetch is only charging the actual Claude crawler if nothing else on the internet has learned to say it's Claude too, and DataDome's numbers say plenty already has.

Signed verification closes that gap where it's been switched on, which so far is a handful of large platforms and the crawler operators motivated enough to register a key. The rest of the conversation, blocking by name, charging by name, allowlisting by name, is still running on the honour system robots.txt was built on thirty years ago. The interesting fight in AI crawler traffic this year isn't really the price per fetch. It's who bothers to prove who they are, and how long the rest of the web keeps trusting a label instead of asking.