We Ran the Standard Crawler Audit on 66 AI Visibility Vendors. All 66 Passed.

Every tool in this market sells a crawler-access check, and they all run the same three: read robots.txt, fetch the page, look for hidden paths. We ran those three against all sixty-six vendors in our own catalogue, ourselves included. Sixty-six passed all three.

That is not a compliment to the category. It is a result about the checks, and we can say so with some authority, because our own domain passed all three on a day when a crawler was being turned away at our CDN.

Disclosure: EchoWi is one of the sixty-six and sells AI visibility measurement, so a study concluding that the free check is uninformative is a study with an interest. The three methods are described in full below, the date is 11 August 2026, the eleven crawler names looked for are listed, and the counts are in our measurement register. Nothing in it spoofs a user-agent, for a reason the method section explains.

The short version

1. Zero of sixty-six block any AI crawler in robots.txt. Eleven agents were looked for, from GPTBot to Applebot-Extended.
2. Sixty-six of sixty-six serve a plain, non-browser HTTP client, which is what most crawlers are.
3. Zero hide pricing, comparisons or documentation from crawlers. The only disallowed paths anywhere are feeds, admin routes and query-string duplicates.
4. Sixteen name AI crawlers explicitly, and all sixteen allow them, which changes nothing, because a crawler nobody mentions is already allowed.
5. A clean sweep on all three is exactly what our own site showed while it was blocking a crawler, and that is the point of the article.
6. No user-agent was spoofed anywhere in this, because sending a crawler’s name from your own machine proves nothing and is how most of these audits are done.

The three checks, and what each one actually asks

Read robots.txt. This asks what the site has declared. It is a published instruction to well-behaved crawlers and it is the only one of the three that is unspoofable, because the file belongs to the site.

Fetch the page. This asks whether a server returns content to a client that is not a browser. We sent a user-agent that says what it is, echowi-accessibility-check, rather than pretending to be GPTBot. That distinction is the whole difference between a measurement and a guess: a fake crawler string sent from our machine is not that crawler, and a CDN deciding on verified-bot status and origin IP will let it through every time. Any audit that tests access by sending someone else’s name is testing nothing.

Look for hidden paths. This asks whether the commercially useful pages, pricing and comparisons and docs, are excluded from crawling.

Those three are what a free AI-visibility audit runs. They are all reasonable and all cheap, which is why every tool includes them.

The results

The eleven agents looked for were GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Bytespider, CCBot and Applebot-Extended.

Nobody declines anything in the other direction either: asked the same sixty-six what they declare rather than what they block, five have a Content Signals directive and all five permit. Re-measured three days later against a larger catalogue, and reading every directive line rather than the first, seven declare Content Signals and one of them declines training, which is also where the instrument error behind the earlier count is written up. The disallowed paths that do exist are housekeeping. Surfer excludes a paginated blog parameter, Semrush excludes tracking-parameter variants and its blog search, MaxAEO excludes comment feeds, Rankability excludes WordPress admin directories, Ahrefs excludes archive queries, Alhena excludes tag pages, SE Ranking excludes a static tools directory and Ceyo excludes some partner embed docs. Not one of the sixty-six excludes a page a buyer would want.

The sixteen who wrote rules that do nothing

Sixteen vendors name AI crawlers explicitly in robots.txt: Airefs, Cloudflare AEO, EchoWi, KIME, MaxAEO, Omnibound, Qwairy, Rank Prompt, Rankability, Rankscale, SE Ranking, ScalePost, Siftly, Superlines, Surfeo and Wellows.

All sixteen allow them. And an explicit allow for a crawler that nobody was blocking changes no behaviour at all, because the default state of an unmentioned crawler is already permitted. Writing User-agent: GPTBot followed by Allow: / produces the same access as writing nothing.

So the only measurable difference between those sixteen and the other fifty is that sixteen went to the trouble of stating an intention. It is a reasonable thing to do, since a future wildcard rule would then not sweep the named crawlers up with it. It is not access, and a checklist that scores it as a point is scoring a comment.

We are in the sixteen. This paragraph applies to us.

Why a clean sweep is the wrong kind of reassuring

Here is the case that makes the three checks look thin, and it is ours.

Our robots.txt allows every AI crawler by name. Our pages are static and served to any client that asks. Nothing is hidden. On all three checks above, our domain scores perfectly, and it scored perfectly on the day we discovered that a major AI crawler was being challenged on almost every request it made to us.

The cause was not in robots.txt and could not have been. Cloudflare’s Bot Fight Mode challenges any bot that is not on its verified list, it runs outside the Ruleset Engine so a WAF rule cannot exempt anything from it, and it decides on verified-bot status and origin IP rather than on the string a client sends. None of the three checks can see any of that. The only thing that could see it was the CDN’s own logs, which are not public, which is why a third party running the standard audit on us would have produced a green result and been wrong.

That generalises uncomfortably. Sixty-six vendors passing three checks tells you sixty-six vendors have not published an instruction to stay away. It tells you nothing about what their CDNs do, and CDN configuration is where the failure we actually experienced lived.

What to ask instead

Ask for the log, not the check. The question that has an answer is which crawlers reached your origin, how often, and with what status codes. That data exists in your CDN or server logs and nowhere else. If a tool tells you your site is accessible to AI crawlers without having read those, it has told you what it could see rather than what you asked.

Cross-reference user-agent with the request path. Attribution by user-agent alone is worthless in both directions. When we looked at what one crawler’s name had been requesting from us, the paths included private key files and credential filenames, which is a vulnerability scanner wearing a crawler’s name and not a crawler. A legitimate crawler asks for articles.

Split the window at the change. If you fix something, do not measure across the fix. An aggregate that spans before and after describes the average of two regimes and neither of them, and it is how a fix that worked gets reported as a fix that half-worked.

And treat a green result as the absence of one kind of problem. Passing these three means you have not told crawlers to leave. That is genuinely worth confirming, it takes a minute, and it is where this class of audit ends rather than where it stops being useful.

And the publishers holding the slots do the same thing, only louder

The check in this piece asks whether a vendor blocks AI crawlers. Run against the publishers that actually hold citation slots, it produces something stranger than a clean sweep: four cited hosts disallow AI crawlers and are cited anyway, one of them disallowing every crawler we checked while sitting second in the answer. That is the sharpest version of what is wrong with this audit as a predictor. It reads a file that is not in the path between a page and an AI Overview.

What this does not show

Nobody was tested for actual access. Not one measurement here establishes that a crawler reaches any of these sixty-six sites, including ours. That would require each vendor’s own logs.

One reading, one day. robots.txt changes, and a vendor could add a blocking rule tomorrow. The eleven agents are the ones that were looked for; an agent outside that list could be blocked and would not appear.

Blocking is not the only failure. A site can be crawled and still be unciteable, which is most of what the rest of this blog measures.

A 403 is not always a block. One vendor’s pricing page returned 403 to our plain client while its homepage returned 200. That is consistent with a bot-management rule and equally consistent with a rate limit or a path-level rule, and we have not distinguished them, so it is not counted as a failure in the table above.

And we are inside the sample. EchoWi is one of the sixty-six and scores the same as everyone else on all three, which is the whole argument rather than a caveat to it.

Common Questions About AI Crawler Access

Does a permissive robots.txt mean AI crawlers can read my site?

No, and this is the most common mistake in this category. robots.txt is an instruction to crawlers that choose to obey it. It says nothing about whether your CDN challenges the request before your server ever sees it, and bot-management products decide on verified-bot status and origin IP rather than on the file. Our own site allows every AI crawler by name and had one of them challenged on nearly every request.

Can I test crawler access by sending the crawler’s user-agent?

No, and doing so produces a false pass every time. A request from your machine carrying PerplexityBot as its user-agent is not PerplexityBot, and any bot-management layer worth paying for decides on the verified-bot list and the origin IP. We ran that test once on ten agents, got ten 200s, and declared the layer clean. It was not. The only thing that settled it was the CDN’s own request logs.

If nobody blocks AI crawlers, why does anyone check?

Because a block is cheap to cause and expensive to miss. A default-deny rule in a bot product, a firewall template, an inherited config from an agency, and a crawler is gone with nothing on the page to show it. Checking is worth the minute it takes. What it is not worth is treating the green result as a statement about your visibility, which is a different problem with different causes.

What is the difference between GPTBot and ChatGPT-User in robots.txt?

They do different jobs and blocking them has different consequences. GPTBot is the training and indexing crawler, and ChatGPT-User is the fetch made when a user’s question causes a page to be retrieved live. Blocking the first is a decision about training data. Blocking the second removes you from answers being written right now. Sixteen vendors in this sample name them separately, which suggests the distinction is understood in this market even where the rules have no effect.

Should I write explicit Allow rules for AI crawlers?

It costs nothing and it grants nothing. An unmentioned crawler is already permitted, so an explicit allow changes no behaviour today. The reason to write them is that it keeps a named crawler out of any future wildcard rule you or an agency might add later, which is a small hedge rather than an improvement. We have written them, and this article is the reason we are not claiming credit for it.

What would a genuinely useful crawler audit look like?

It would read your CDN or server logs rather than making requests from somewhere else, report by crawler and by status code, cross-reference the user-agent with the paths requested so scanners wearing crawler names are separated out, and split any window at the moment a configuration changed. Every one of those needs access to your infrastructure, which is why the free version of this check does the three things above instead, and why the three things above return sixty-six clean results.