The First AI Visibility Vendor to Block a Crawler Did Not Type the Rule

On 17 August 2026 we swept this category’s robots.txt files: 68 vendors served one and 0 disallowed any of the 14 AI crawlers we track. We ran the same sweep on 21 August over a larger catalogue. 77 serve a file and 1 blocks. The interesting part is not the count. It is that the blocking vendor did not type the rule, and that what the rule shuts out is training rather than retrieval, splitting our 14 crawlers into exactly 7 blocked and 7 allowed.

Disclosure and method: EchoWi sells AI visibility measurement and every vendor in this sweep is a competitor. That is why the method is mechanical rather than editorial: we fetch each vendor’s robots.txt at its own host, evaluate the rules for 14 named crawlers at the path each file states, and report what the file says. No user-agent is spoofed, because a spoofed agent proves nothing about access. 78 catalogue entries asked, 77 served a file, 1 behind a bot challenge and put in its own bucket rather than counted as compliant, 21 August 2026. The counts are in our measurement register, and the file is public, so anyone can repeat this against us.

The short version

1. A published zero is gone. On 17 August 2026, 68 vendors served a file and 0 disallowed any AI crawler. On 21 August, 77 serve and 1 blocks.
2. The rule is a managed default, not the vendor’s sentence. The file carries # BEGIN Cloudflare Managed content and every block sits inside it. After that section closes, the vendor’s own contribution is a single Disallow on one path.
3. It blocks training and leaves retrieval open. Of our 14 crawlers, 7 are disallowed and 7 are not, and the line between the halves is what the crawler is for.
4. The crawlers that carry citations are all in the allowed half, so on this evidence the vendor loses corpus inclusion and keeps answers.
5. The managed list is not exhaustive, and the gap is named below.
6. This previews 15 September, which is the reason to care and is in the last section.

Which vendor, and why we name it

The vendor is Knowatoa, and knowatoa.com/robots.txt says all of this in public. Apex and www serve byte-identical 1,968-byte files, checked separately, because a host and its www can differ and this register has already been caught once by assuming they cannot.

We name it for the same reason we publish the exact prompt behind every measurement: a figure you cannot check is not a figure. Anyone can fetch that file in one command and see whether we read it correctly.

We are also going to be careful about what it means, because the obvious headline is wrong. “An AI visibility vendor blocks AI crawlers” is a good line and a bad description. The block sits inside a managed section, which is a setting rather than a sentence somebody wrote, and it stops the crawlers that build training corpora while leaving alone every crawler that fetches a page in order to answer a question.

What the file actually contains

Three parts, in this order.

A preamble. Twenty-six lines defining the Content Signals vocabulary, ending in a declaration that any restriction expressed through it is an express reservation of rights under Article 4 of the EU copyright directive. This is boilerplate shipped with the managed file, not something the vendor drafted, but it is doing legal work regardless.

The managed block. It opens with a * group carrying Allow: / and Content-Signal: search=yes,ai-train=no,use=reference, then disallows nine agents outright: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot and meta-externalagent.

The vendor’s own rule, after # END Cloudflare Managed Content. It is one line: a second * group with a Disallow on a single path of the site.

So the vendor’s own crawler policy is one path. Everything that reads as a stance on AI is a managed list, and the Content-Signal line says the same thing in the other vocabulary: indexing and excerpting are fine, training is not.

There is a structural wrinkle worth noting for anyone auditing files like this. There are two * groups, one in each section. A parser that stops at the first match sees Allow: / and reports the site as fully open; a parser that stops at the last sees only the path rule. Neither is the whole file. We have made this exact mistake before, on somebody else’s file, and it is why our sweep reads every group rather than the first one it finds.

Training and retrieval are different permissions, and only one is blocked

This is the distinction the category keeps collapsing, and this file draws it about as cleanly as it can be drawn.

That is 7 and 7. Every crawler in the left column exists to collect corpora. 6 of the 7 in the right column exist to fetch a page while answering somebody’s question. A brand that blocks the left column loses training inclusion and keeps citations. A brand that blocks the right column disappears from answers.

That is why the raw count would mislead. Half the tracked crawlers blocked sounds severe. Half the tracked crawlers blocked, where none of the blocked half carries a citation, is a different fact and points the other way.

The file also blocks Amazonbot and CloudflareBrowserRenderingCrawler, which are outside the 14 we track and so are not in that table. We mention them because leaving them out of the prose would make the file look smaller than it is.

The gap in the managed list

anthropic-ai sits in the allowed column, and it does not belong there on the logic of the rest of the file. It is the older agent name Anthropic used for training collection. The managed list blocks ClaudeBot and misses it.

That is not a criticism of the vendor, who did not write the list. It is the argument against treating a managed block as a policy: it is a snapshot of one provider’s view of which agent names matter, it is maintained on somebody else’s schedule, and a name that falls outside it is simply not covered. If the intent is ai-train=no, this file does not fully deliver it, and nothing in the vendor’s own site would tell them.

Did it change, or did we miss it

This is where we have to be careful about what we can actually prove.

The vendor entered our catalogue on 7 August, ten days before the sweep that found 68 vendors serving and none blocking. The single vendor excluded from that sweep was behind a bot challenge and is a different company. So the inference is that this file changed between 17 and 21 August 2026.

It is an inference, not a stored observation. That sweep recorded aggregates, 68 served and 0 blocking, not a row per vendor. There is no line in our register that says this vendor allowed everything on 17 August. What we have is a total with no room in it for a blocker, and a catalogue that already contained this vendor. That is sound, and it is weaker than a diff.

The fix is the obvious one and it is now the plan: store the per-vendor result rather than the total, so next time a change is a diff instead of a deduction. We are writing that down publicly because the same failure is probably sitting in your own monitoring. An aggregate is enough to notice that something moved and never enough to say what.

Why this matters more on 15 September than it does now

Cloudflare changes its defaults for AI bots on 15 September 2026. Two parts of that bear on this file.

The first is that the change applies to the managed rules, which is the section doing all the blocking here. Nobody at the vendor has to touch anything for the behaviour of their site to change.

The second, and the one worth writing down, is that crawlers doing search and training at once become blocked by any setting that blocks training, including settings chosen before the change.

For a company that sells visibility in AI answers, that is the expensive direction. A training block is cheap: you lose corpus inclusion and keep citations, which is a coherent trade. A training block that also catches the mixed crawlers is not cheap, because the mixed crawlers are the ones that fetch a page in order to cite it.

Nothing here says that has happened. Today’s file blocks training and leaves retrieval open, which is exactly what a company that wants to be cited and not absorbed should want. The point is that the position is currently held by a default, and the defaults move on 15 September 2026.

What this does not show

One vendor is not a trend. The honest summary is that a zero became a one on a population that also grew. The content is the mechanism, not the rate, and we would say the same if the count went back to zero next week.

A declaration is not a behaviour. A robots.txt states what a crawler is asked to do. It does not establish that the crawler obeyed, that the pages are cited, or that anything changed in any answer. Every crawler-access figure we publish carries this limit and this one is no different.

We cannot prove the file changed, only that our own aggregate from four days earlier leaves no room for it.

We did not ask the vendor. This is a reading of a public file, not a report of anyone’s intent. A managed block is a setting, and a setting can be on for reasons that have nothing to do with a position on AI.

And the other 76 are the finding underneath this one. 76 of 77 serve a file that disallows none of the 14 crawlers we track. Whatever this category argues about in public, its own front doors are open.

Frequently asked questions

Is blocking training crawlers a mistake for a brand?

Not on its own, and the trade is legible. Blocking GPTBot, ClaudeBot and Google-Extended removes you from training corpora while leaving OAI-SearchBot, Claude-User and Googlebot free to fetch your pages when an assistant answers a question. If your worry is being absorbed and your goal is being cited, that is the coherent setting. It becomes a problem when a rule meant for training also catches the crawlers that retrieve, which is what changes on 15 September 2026.

How do I check my own file for this?

Fetch yourdomain.com/robots.txt and look for # BEGIN Cloudflare Managed content. If it is there, the block list inside it is not yours and it will move when the provider’s defaults move. Read every group rather than the first one, because a managed file and your own rules can both target *. Check the apex and the www separately, because they can serve different files.

Can a managed robots.txt replace the one I wrote?

Yes, and that is the risk worth knowing. The managed file can be served in place of yours, so a declaration you committed can stop being served without anything in your repository changing. We audit our own setting for exactly this reason, because for a company that publishes crawler declarations, having them silently replaced would be the most embarrassing failure available.

Why a crawler list and not one number?

Because training and retrieval are separate permissions and a single count hides the difference. The list covers both sides for each major provider, so a file that blocks one and allows the other shows up as such instead of collapsing into a score. This file is the case in point: one number would have read as half blocked, when the useful fact is which half.

Does Content-Signal do anything on its own?

It is a declaration, not an enforcement, and it is young. It is worth setting because it states intent in a machine-readable way and, per the preamble shipped with these files, is framed as a reservation of rights under EU copyright law. What it does not do is stop a crawler. The Disallow lines are what a compliant crawler obeys, and neither one binds a crawler that ignores the file.