What 2.7 Million AI-Crawler Log Rows Reveal About How Bots Read the Web

Almost everything written about AI crawlers is guesswork. People reason from a bot's published documentation, or from a handful of log lines, and extrapolate. That's understandable - most teams never see their crawler traffic at the source. But it produces confident claims about bot behavior that real data quietly contradicts.

We took a different route: we looked at 2.7 million rows of real AI- and search-bot access traffic, captured at the CDN edge where every request actually lands. Not modeled, not sampled from a few pages - real requests. This is less a single dramatic statistic and more a lesson in why the source of the data changes the conclusions you can draw. Here's what that kind of dataset reveals, and why it matters for anyone trying to win AI search.

Why CDN-level data is different from everything else

First, the methodological point, because it's the whole reason this is worth reading. There are three ways to "know" what AI bots do on your site, and they're not equal:

• Documentation: what a crawler's operator says it does. Useful, but aspirational and incomplete.
• Sampling / estimation: inferring behavior from a slice of pages or third-party panels. Directional, but blind to your specific site.
• CDN-level logs: the actual requests hitting your edge - every bot, every page, every fetch. Ground truth.

The first two are where most "AI crawler" content comes from, and they systematically miss the things that matter most: the bot that's silently blocked, the key page crawlers never reach, the difference between a bot that visits once and one that returns daily. You can't estimate your way to those - you have to observe them. That's what a dataset measured in millions of real rows lets you do.

What this kind of data actually surfaces

When you analyze real crawler traffic at this scale, a few categories of insight emerge that estimates can't give you:

Which bots actually show up - and which don't. Operators publish many crawler user-agents, but the ones that genuinely hit a given site (and how often) vary enormously. Real logs separate the crawlers that matter for your visibility from the long tail that barely appears.

The purpose mix: cite, index, or train. Not all bot traffic is equal. Some requests are live retrievals feeding an answer (cite), some build search/answer indexes (index), and some gather training data (train). The blend differs by site and tells you very different things - a site getting frequent cite-traffic is in the live answer flow; one seeing only train-traffic is being learned from, not currently quoted.

Crawl frequency and depth. How often bots return, and how deep they go, reveals whether your important pages are actually being seen or whether crawlers stop at the homepage. Depth-vs-discoverability patterns are where "we published it" and "a bot actually fetched it" diverge.

What's being skipped - and what's wasting budget. Real logs show the pages bots ignore (often the high-value ones, behind a render or a rule) and the low-value pages soaking up crawl attention. Both are invisible without the data.

The accidental blocks. The most actionable finding in almost any real crawler dataset is the same: bots that should be reaching key content and aren't - quietly blocked by a rule, a header, or a render dependency nobody remembered. You only catch these by watching real traffic.

We're deliberately not publishing invented percentages here, because the honest value of this dataset isn't a single number to quote - it's the capability it represents: seeing your non-human visitors as they actually behave, on your actual site, instead of guessing.

Why it matters for your AI visibility

Your most important visitor is no longer human; it's a crawler deciding whether to cite, index, or train on your content. If you can't see what those crawlers do, you're optimizing for an audience you've never observed - and the failure modes are silent. A blocked answer-bot doesn't show up as a traffic dip you can trace; it shows up as an absence in AI answers you'll never connect to its cause.

That's the case for measuring bot behavior at the source rather than reasoning about it from the outside. The brands that win AI search will be the ones who treat AI crawlers as a measurable audience - who can say "the bots that feed answers reach our key pages this often, and here's the one that's being blocked" - instead of the ones working from a crawler's documentation and hope.

How to see your own crawler reality

You don't need millions of rows to start - you need to look at the source. Your CDN or server logs already contain your real bot traffic; the work is identifying the AI crawlers, classifying them by purpose, and watching for the silent blocks and the unreached pages. Done by hand it's tedious; done continuously it's the difference between guessing and knowing.

That's exactly what our BotsGenie module is for: a lightweight Cloudflare Worker at your edge that identifies every AI crawler, classifies it as cite / index / train, and surfaces the patterns - and the accidental blocks - that estimates miss. (See BotsGenie and, for the access side, which AI crawlers can access your site.)

The first step is the same one we took: stop guessing what bots do, and look at what they actually do. Run a free AI visibility report to check the access and readiness side, then watch the real traffic from there.

Frequently asked questions

Why not just read the AI crawlers' documentation?

Documentation tells you what a bot is meant to do, not what's happening on your site. Real logs catch the gap - the blocked bot, the unreached page, the actual frequency - which is where the actionable findings live.

What's the difference between cite, index, and train traffic?

Cite-bots fetch your page live to use in an answer; index-bots build the databases AI draws from; train-bots gather data for models. The mix on your site tells you whether you're in the live answer flow, merely indexed, or only being learned from.

Do I need a huge dataset to learn anything?

No - you need the right source. Even your own CDN/server logs reveal your real crawler behavior. Scale helps spot broad patterns, but the highest-value finding (an accidental block) shows up at any size.

Why can't estimate-based tools catch accidental blocks?

Because a block is the absence of a request, and you can't estimate an absence from a sample of other sites. You can only see it by observing your own real traffic, where the expected request simply never appears.

What do I do once I find a problem?

Fix the cause - unblock the crawler, fix the render, correct the path - then confirm the bot returns. Observation pairs with remediation: see it, then fix it.

The web is full of confident claims about how AI crawlers behave, almost all of them estimated. Real CDN-level data - measured in millions of requests - tells a more useful and less tidy story: which bots actually show up, what they're really doing, what they skip, and which key pages they've been silently blocked from. You can't optimize for an audience you've never watched. Start watching the one that now decides your AI visibility.

Check your access and readiness: run a free AI visibility report.