The Crawler Takes Your Front Door and the Assistant Takes Your Articles. One Day in Seven, Neither Does.

Everyone in this category argues about whether the AI crawlers can reach you. Almost nobody has looked at what they do once they arrive. We read a week of our own edge logs grouped by user agent and by path, and the agents split into two groups that behave in opposite ways.

The crawlers take your front door and the assistants take your articles. Across 57 readings of 13 agents over 7 days, the assistant group spends a median 77% of its fetches on content and 8% on the home page. The crawler group inverts it: a median 4% on content and 45% on the home page.

And then one day in seven erases the distinction completely, which is the part of this that transfers to your site whether or not our split does.

Disclosure and method: EchoWi sells AI visibility measurement, so this is a measurement of our own category made with our own data. The source is the edge log of our own domain for 13 to 19 August 2026, grouped by request path and requesting agent, with page type decided by the URL rather than by any judgement about intent. A reading is one agent on one day; readings under a floor of twenty requests are kept in the register and excluded from every share, because a percentage of a handful is noise. Only shares appear below and the denominators are withheld on purpose. Anyone with a CDN log can run this query against their own site and check whether the shape repeats. The rows are in our measurement register.

The short version

1. Crawlers and assistants are not the same visitor. They come from the same companies and go to opposite places.
2. The split is a median, not a rule. 5 crawler readings put half or more of their fetches on articles, the highest at 77%.
3. One day in seven collapses it, and it is the day with the most traffic in the week.
4. Not one agent fetched a markdown twin all week, while 11 of them fetched the machine-readable text files.
5. The single biggest destination for the largest assistant is our reviews of other companies’ tools.

What the week says

Percentages of each agent’s fetches that landed on an article, a vendor review, a study, a glossary term or our measurement register. Days run 13 to 19 August; a gap is a day whose reading fell under the floor.

Read the rows rather than the medians and two things are obvious. Claude-User never drops below 59%. OAI-SearchBot never rises above 12%. Those two are the same shape every day of the week, from different companies, and neither has anything to do with our configuration.

The correction, which is the whole reason to widen a study

We published this two days into it. That version reported the assistant median at 58% content and the crawler median at 2%, and it drew the same conclusion.

The week says 77% and 4%. The first reading understated the split rather than inventing it, which is the comfortable direction for us and still a correction. Two days of anything is a shape seen twice.

It also killed a sentence. The two-day version said three crawler readings fetched no article at all, which is true and which a reader can easily hear as crawlers do not fetch articles. Over the week that is false: 5 crawler readings put half or more of their fetches on articles, and GPTBot spent one day at 73% with a volume well inside the usable range. A crawler fetching your articles is not rare, it is a minority behaviour with a long tail, and there is a real difference between those two claims for anyone deciding what to allow.

The day the split disappeared

On 13 August, seven agents went to the home page at once. ChatGPT-User, which spends 71 to 92 per cent of its fetches on articles on every other day of the week, spent 10% on articles and 84% on the home page, at a volume roughly an order of magnitude above its weekly norm. GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot and Amazonbot all sat near 90% home the same day. It was the highest-traffic day of the week on this domain.

The following day every one of them was back to its usual shape.

We are not going to explain it, because we cannot. A coordinated re-crawl and a burst of traffic wearing agent names produce exactly the same rows, and a user-agent string is a claim rather than a proof. What matters is not the cause:

Anyone who had read a single day of logs on 13 August would have concluded that all AI agents behave identically and all of them only want your home page. They would have been reading the one day in seven where that is what the data says. That is the practical warning in this study, and it is stronger than the split itself, because it applies to your logs whatever your split turns out to be.

Nobody fetched the markdown

We publish a machine-readable twin of every page: llms.txt, llms-full.txt, and an index.md beside each index.html that content negotiation will also serve under Accept: text/markdown. That is the setup this whole category recommends.

In a week, across 57 readings, no agent fetched a single markdown twin. Not one.

The text files are a different story. 11 of the 13 agents fetched the machine-readable files, and one reading spent 44% of its fetches on them. OAI-SearchBot, Claude-User and Bytespider are the heaviest, and Claude-User in particular spends between 9 and 41 per cent of its day on them depending on the day.

So the machine surface is being read, and the specific convention that gets the most enthusiasm is the one nothing touched. Our own conclusion is not to remove it: the file costs nothing to generate, one week is one week, and an agent that never asks today may ask next quarter. Our conclusion is that if somebody is selling you markdown twins as an AI visibility lever, ask them for the fetch log. We then asked the obvious next question of the people selling it, and 12 of 77 vendors in our catalogue serve markdown at all while 63 answer the request with HTML.

A robots.txt says what is allowed. This says what happened

Most of what gets written about AI crawling is a reading of declarations. You open a robots.txt, you see which agents are named, you conclude something about access. We have run that measurement ourselves across two populations and it is worth running, but it answers a narrower question than it appears to.

A declaration is a statement of policy. It cannot tell you whether the agent obeyed it, whether it arrived at all, or what it took. Those are three separate facts and only the log has the last two.

Here they diverge in a way that matters. Our site allows every one of these agents, identically, with no rule separating one from another. Under an identical policy the agents produced two opposite itineraries, and on one day a third. Whatever decides that, it is not our configuration, because our configuration says the same thing to all of them.

What a site owner can do with this

We have since asked whether the three assistants read the same pages as each other, and their page sets turn out to be nested rather than different: nearly everything the smaller ones fetch, the largest one fetches too.

Stop treating one number as your crawl coverage. If a dashboard tells you AI bots fetched N pages this week, that number sums two populations doing opposite things, plus whatever the anomalous days contribute. Split by agent before you read it. That is one extra dimension in the query.

Never read one day. Our worst day is not a bad reading, it is a plausible one that says the opposite of the week. If your log tool shows you yesterday by default, that is the number you will act on.

Look at which of your pages the assistants take. Ours take our reviews of other companies, which is a finding we did not want and could not have got from any keyword tool. Whether that set is the same one search already shows you is a separate question, and answering it needs a second instrument on the same domain. Yours may take your documentation, your pricing, or one comparison post from a year ago. That list is one query away in your own logs and it is in nobody’s keyword planner.

What this does not show

One site. This is a shape, and a shape seen on one domain for one week. A second site would tell you whether the split is general or ours.

A user-agent string is a claim, not proof. Every figure here is what identified itself as that agent. You cannot prove crawler access by sending a user agent and you cannot attribute traffic by reading one, and this measurement inherits that ceiling. What makes the shape credible rather than the volume is that the two groups differ from each other in a way a spoofer has no reason to reproduce, and that the paths are articles and front pages rather than the credential files a scanner asks for. The 13 August anomaly is exactly where that ceiling bites, and we have said so rather than picking the explanation we prefer.

Page type is decided by the URL. A review is a path ending in -review/. Anyone can apply that rule to our site and check it, and it will misclassify any page whose URL does not describe it.

Being fetched is not being cited. This study measures only the first of those. The pages fetched most often on our domain are the ones least likely to name us in the answer that follows, because they are about somebody else.

And the absolute numbers are withheld deliberately. We publish measurements about the market and not our own traffic figures, in either direction. The compensation is the method: the query is three lines and your own logs will answer it.

Common Questions About AI Agent Crawling

Do AI crawlers actually visit small sites?

On these seven days, 13 distinct AI agents did, from at least six companies, and they arrived every day rather than in one burst. The more useful question is not whether they arrive but which of them arrives, because the crawler that indexes you and the assistant that answers with you are different visitors with different itineraries.

What is the difference between GPTBot and ChatGPT-User?

GPTBot crawls for training and indexing on its own schedule. ChatGPT-User fetches a specific URL because someone asked a question and a retrieval step chose that URL. Over our week ChatGPT-User spent a large majority of its fetches on articles on six days out of seven, while GPTBot spent almost none on five of them. Blocking one and allowing the other are completely different decisions.

Should I block AI crawlers?

That is a business decision and this measurement does not make it for you. What it does say is that blocking is not one switch: the agent that could cite you tomorrow and the agent that trains on you tonight arrive under different names, and a rule written for one applies to the other only if you write it that way. We measured separately that the category selling AI visibility blocks none of them, while the platforms that advice tells you to publish on block several.

Is llms.txt worth publishing?

The text files do get fetched: 11 of the 13 agents asked for machine-readable files at some point in the week, and one reading spent 44% of its day on them. The per-page markdown twins are the part nothing touched. We are keeping both, because generating them is free and one week is a short window, but we would not describe either as a measured visibility lever, and neither should anyone selling them to you.

Why would an assistant fetch a review of a competitor?

Because the question it is answering is probably about that competitor. A page comparing tools is a candidate answer for a comparison question, and comparison questions are most of the commercial demand in this category. It is also why the biggest single destination in our own logs is a set of pages about other companies.

How do I run this on my own site?

Group your CDN or server logs by user agent and by request path for a single day, keep the rows whose agent names a known AI agent, and bucket the paths by what kind of page they are. Then do it for six more days, because the whole point of this study is that one day lies. The finding is the gap between two columns: the share landing on your home page and the share landing on your articles.

Does being fetched mean being cited?

No, and the gap between the two is measurable. The pages fetched most often on our own domain are the least likely to name us in the answer that follows. Fetching is the first step of a chain and the only one this study observes.