We checked 26 publishing hosts against 14 AI crawlers. Eleven let Google in and shut the AI out.

The advice is everywhere right now: publish your page on a big domain you do not own, and it ranks in hours instead of months. The newest version of it uses published Claude artifacts and is sold explicitly as an AI-era tactic. We read the robots.txt of 26 hosts against 14 crawlers, and then doubled the list and read it again. Eleven of them admit Googlebot and refuse at least one AI crawler, and the two AI companies that let you publish a page block each other. And of the 68 vendors selling AI visibility, 0 block any of the 14.

Disclosure: EchoWi sells measurement of AI visibility, so a study concluding that a free tactic does not deliver AI visibility is a study with an interest in its own result. The answer to a conflict is a method you can repeat: every host, the exact path, the 14 crawler names, the date, and the user-agent group that decided each verdict are all below and all in our measurement register. The vendor side is our public catalogue, so both populations are lists you can rebuild. Every one of them is a public file you can fetch yourself in a second.

Update, 21 August 2026. The vendor half of this study was re-measured after the catalogue grew, and the zero above did not survive: 77 vendors serve a file and 1 now disallows 7 of the 14 crawlers. The 68 and the 0 stay exactly as written, because they are what the 17 August sweep found and a frozen figure must not follow a later measurement. What changed, and why that block is a managed default rather than a rule the vendor typed, is in the follow-up.

The short version

1. 26 hosts served a robots.txt. Two did not and are reported separately: one answered a bot challenge and one serves no file at all. Blocked is not missing, and missing is not permission.
2. Eleven hosts admit Googlebot and disallow at least one AI crawler. That is the gap a buyer cannot see from outside, because the page ranks and cannot be read.
3. The AI companies block each other. Four of these hosts let a user publish a page. Two of the four disallow their competitors’ crawlers and allow their own.
4. Hosts block training more than they block search. GPTBot is refused by 6 and OAI-SearchBot by 4. ClaudeBot by 8 and Claude-User by 4.
5. Googlebot gets into 25 of 26. The single exception disallows everybody, Google included.
6. And there is a second mechanism that has nothing to do with robots.txt. For the tactic that started this, the page content is not in the HTML at all.
7. The vendors selling AI visibility block nothing. Of the 68 that served a file, 0 disallow any of the crawlers. Of the hosts this advice names, 12 of 26 do.
8. We doubled the list and the proportion held. 9 of the next 25 admit Googlebot and refuse an AI crawler, against 11 of 26 in the first sweep.

Why we checked instead of arguing

The tactic is old and the framing is new. Publishing on a domain with more standing than yours has been a search tactic for two decades. What changed is that it is now being recommended for visibility in AI answers, which is a different claim about a different set of machines, and nobody appeared to have checked whether those machines are allowed in.

That check costs one HTTP request per host. It seemed worth doing before writing anything about whether the tactic works.

What we measured, and what we did not

One robots.txt per host, fetched twice on 17 August 2026, parsed into groups, and judged at the path the tactic actually uses. So claude.ai is judged at /public/artifacts/, chatgpt.com at /share/, linkedin.com at /pulse/, and the rest at the root.

The parsing is the whole difficulty and it is where a quick answer goes wrong. A robots.txt is a list of groups. A group is one or more consecutive User-agent: lines that share the rules below them. A crawler obeys the group whose token is the longest match for its own name and only falls back to * when no named group matches. So a file whose * group allows everything can still disallow GPTBot two groups further down, and a reader that stops at * reports the exact opposite of the truth.

Both readings agreed on every host and every crawler, with no differences at all, which is the stability control this kind of sweep needs and rarely gets.

What this does not measure, stated here because it is the honest boundary: whether the crawlers obey the file, whether those pages get cited in AI answers, and whether ranking follows. A declaration is a declaration. What a file says and what an edge does are different layers, and the second needs server data from a domain you control. We have since read that second layer on our own domain, and it says the agents split into two groups that fetch completely different pages under one identical policy. It also does not make the vendor comparison causal: two populations read the same way on the same day describe each other, they do not explain each other.

The 14 crawlers are OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User; Google’s Google-Extended and Googlebot; Anthropic’s ClaudeBot, anthropic-ai and Claude-User; PerplexityBot and Perplexity-User; and CCBot, Applebot-Extended, Bytespider and meta-externalagent. Training, search and fetch-on-behalf-of-a-user are three different permissions, and keeping them apart turned out to matter.

The AI companies block each other

Four of the hosts in this sweep now let a user publish a page that anyone can read. Two of them disallow their rivals’ crawlers.

The two blocking files do it differently and arrive at the same place. Anthropic stacks four agent names above a single Disallow: /, which is why a parser that assumes one agent per group reads it as blocking one crawler and allowing three. OpenAI writes one group per competitor. Both allow their own crawler, and both allow Googlebot.

If you publish a page on either of those two hosts in order to be read by AI assistants, the assistant most likely to read it is the one belonging to the company whose domain you published on.

Googlebot gets in nearly everywhere, and the exception is the interesting one

Googlebot is disallowed by exactly one host of the 26, and it is not one of the AI companies. It is Reddit.

Reddit’s entire robots.txt is 538 bytes. After a comment pointing at its public content policy, it says User-agent: * and then Disallow: /. That is the whole file. It disallows every crawler in this study, Googlebot included.

Reddit content is obviously in Google. It is there because of a commercial arrangement between two companies, not because the file grants permission. So anyone told to publish on Reddit to get picked up by AI search is relying on a private agreement they are not party to, on a domain whose public instruction to every crawler is to stay out.

Hosts block training more than they block search

This is the finding we did not expect, and it is the one that says the field is more sophisticated than it is usually given credit for.

Read the pairs. OpenAI’s training crawler is refused by 6 hosts and its search crawler by 4. Anthropic’s is refused by 8 and its user-fetch agent by 4. The same ordering holds for Perplexity.

These hosts are not blocking AI. They are drawing a line between being trained on and being searched, and letting search through more often. It is a coherent position and it is visible in a public file, and it means that “does this host block AI crawlers” is the wrong question. The right one is which of the three permissions you need.

For a brand publishing a page to be found, the permission you need is search. That is the one most likely to be granted, which is the small piece of good news in this study.

The vendors selling this all let the AI in

One more sweep, the same day, the same 14 crawlers, the same instrument. We pointed it at the 69 tools in our own verified catalogue, the companies whose product is AI visibility.

Of the 68 that served a robots.txt, 0 disallow any of the 14 crawlers. Not one. And they are not files without rules: Semrush carries 60 Disallow lines, Ahrefs 26 and HubSpot 356. They have plenty of path rules and no AI crawler policy at all. The sixty-ninth answered a bot challenge and is excluded rather than counted as permission.

Put the two populations beside each other and the shape of the advice becomes visible.

The category that sells this practises what it preaches. The platforms it sends you to publish on do not.

That contrast is worth stating carefully, because it is descriptive and not causal. These are two populations with different purposes: a vendor’s own site exists to be found, and a publishing platform has users, rate limits and licensing deals to think about. What the two counts share is the day, the crawler list and the instrument, which is the only reason they can be compared at all.

And two of them publish a homepage no crawler can read

The contrast above is about what a file permits. There is a second permission nobody writes down: whether your words are in the response at all. A page whose text arrives only after JavaScript runs is unreadable to every crawler that does not render, whatever robots.txt allows.

We pointed the same measure at the same catalogue, one homepage each. 67 were read and 2 answered a bot challenge, so those two are reported unread rather than counted either way.

1 of the 67 serves none of its own text anywhere. Bluefish AI answers with 114 bytes: a script that sends the browser somewhere else, and nothing else. No heading, no sentence, no title, no description, no structured data, not even a link for a crawler to follow.

A second page is a different thing, and we first published it as the same thing. GEO-Vantage answers with 4,870 bytes whose body carries nothing, and we counted that as unreadable. It is not. Its head carries a title, a description and structured data, so a retrieval layer has plenty to work with; it simply does not use the body for it. We published this as 2 of 67 and the correct count is 1, with a second page carrying its content in the head. Neither offers a noscript fallback, and both returned byte-identical responses on two readings the same day.

For scale, the next thinnest homepage in the catalogue carries 446 characters of body text and the median carries about ten thousand.

So the category’s record depends on which axis you ask about. On robots.txt it practises what it preaches. On rendering, two companies selling AI visibility publish a front door an AI crawler cannot read, which is a one-line fix and a single request to check.

Every page an AI engine cited was readable, and a third keep it in the head

Both counts above are populations we chose. This one is not. Our register holds 79 URLs that an AI engine actually cited in one of our studies, so the engines picked the sample. 64 could be read, 14 answered a bot challenge and 1 did not answer at all.

0 of the 64 were unreadable. Not one cited page failed to carry its own content somewhere in the response.

But 18 of the 64 carry it only in the head, and every one is YouTube or Facebook. A YouTube watch page answers with about 1.4 MB whose body is 217 characters of site footer and not one word of the video. Its head carries the title, the description, the keywords and a full VideoObject in JSON-LD.

That is worth sitting with, because YouTube is the most cited domain in this whole corpus. The body is not where citability lives for the domain that wins most often. What a retrieval layer needs is the answer somewhere machine readable, and the head counts as somewhere.

What this cannot show, and the design is the reason: the sample is selected on the outcome. “Cited pages are readable” cannot mean rendering causes citation, because pages that get cited may be readable for reasons that also make them worth citing. What a design like this can do is kill the opposite claim. If a large share of cited pages had been unreadable, then serving your text plainly would not be a precondition for being cited. Zero of sixty-four were.

We doubled the list, and the proportion held

The number above counts hosts, so it is a claim about our sample as much as about the web. We asked 32 more, on a later day, with the same 14 crawlers and the same instrument, and the prediction went in writing before the first request: the finding holds if the new hosts land between 30 and 55 per cent.

25 of the 32 served a file. 9 of the 25 admit Googlebot and refuse at least one AI crawler, against 11 of 26 in the first sweep.

Pooled, that is 20 of 51 hosts. Two independent samples of the platforms this advice names, and the gap sits either side of four in ten both times.

Googlebot admission held as well, 24 of 25 against 25 of 26 in the first sweep, and the single exception is the host two paragraphs down.

The second sweep added two things the first could not.

On 5 of the 32 there is no robots.txt at all. That is the permissive extreme rather than a refusal: with no file, nothing is disallowed to anybody. It is counted apart from the 2 that answered a bot challenge, because unread and unrestricted are different states and only one of them is permission.

And one host closes its own published pages to everybody. grok.com names GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Applebot-Extended in a single group and disallows them across the site, which is the pattern from the section above. Then /share returns 404 and /share-links returns 200, so its public pages live on the one path the * group refuses, and at that path all 14 crawlers are disallowed, Googlebot included. It is the first host here where a page you publish is closed to everything, so the tactic fails at the search step and not only at the AI step. We found it by asking which path the pages are actually on instead of measuring the root.

The AI arm was doubled too. 5 more publishing hosts served a file and 1 of the 5 shuts a rival out, so the finding two sections up stands at 3 of 9 across both sweeps. That is a third rather than a half, which is the direction a small sample usually moves when you enlarge it, and the reason to enlarge it.

The second mechanism, which has nothing to do with robots.txt

The tactic that prompted all this is publishing a Claude artifact. So we fetched published artifacts as an ordinary client that does not execute JavaScript.

Every artifact returns the same thing. Byte for byte the same thing: 91,774 bytes, on two different artifact ids, across three user agents. That payload contains a fallback title of Claude Artifact, a noindex, nofollow meta tag, and 78 characters of visible text once scripts and tags are stripped. None of those 78 characters belong to the artifact.

Sending a Googlebot user agent changes nothing. The byte count is identical, so there is no separate prerendered version for search engines. Google has these pages indexed with their real titles because Googlebot executes JavaScript and reads the page the browser would build.

This is the part that matters, and it is independent of everything above. A crawler that ignored the robots.txt entirely would still get 78 characters of chrome. The tactic fails for AI visibility twice over, by two mechanisms that do not depend on each other: the file says no, and the page is empty to anything that does not render.

We are describing how a single-page application behaves. It is not a criticism of the product, which is not built to be a publishing platform for other people’s marketing. It is a description of what a fetcher receives.

If you were about to do this

The tactic is not fake. Those pages are indexed, in volume, in many languages, and some of them are plainly commercial. If your goal is a fast-ranking page in Google web search for a low-competition term, it does what it says.

It is being sold as something else. Ranking in Google web search and being readable by AI assistants are two different outcomes with two different gatekeepers, and this sweep shows they come apart on 11 of 26 hosts.

Check the file before you publish, not after. It takes one request. Fetch https://thehost.example/robots.txt, find the group naming the crawler you care about, and read the rule. If there is no named group, the * group applies. If the page you are publishing is rendered client-side, fetch it with something that does not run JavaScript and see whether your words are in the response.

And prefer a domain you control. Everything above is a decision someone else makes about your content, can change without telling you, and has commercial reasons to change. On your own domain the file is yours.

Limits

• One reading of one file per host, on one date. A robots.txt is a fact of a day. Each row carries its byte count so a changed file shows up as a changed row next time.
• Declaration, not behaviour. Nothing here measures whether a crawler obeys, whether the page is cited, or whether ranking follows.
• 51 hosts across two sweeps, not the web. They were chosen because they are the ones this advice actually names. A different list would give different proportions, so the count is stated everywhere a proportion is.
• The rendering finding is about one platform. We fetched published Claude artifacts. Other hosts in the list render differently and were not tested that way.
• 9 hosts produced no verdict and are excluded from every count rather than folded in as permission.

Common questions about publishing on someone else’s domain

Does publishing on a high-authority domain still work for Google?

Yes, for the narrow thing it does. Those pages are indexed and some rank for low-competition terms. What this study measures is a different question: whether the machines that answer questions, rather than the one that lists links, are allowed to read them.

Which crawler should I actually care about?

The one matching the outcome you want. Training crawlers like GPTBot and Google-Extended decide whether your text can inform a model later. Search crawlers like OAI-SearchBot decide whether it can be retrieved and cited now. User agents like ChatGPT-User and Claude-User fetch a page because someone asked. For being found in an answer today, search is the one that matters, and it is refused least often.

If a host blocks a crawler, does that crawler definitely stay out?

No, and this study does not claim it. A robots.txt is an instruction, not a wall. What it establishes is intent, which is worth knowing before you build a strategy on a domain, and separately from whether it is honoured.

Why does Reddit disallow Googlebot when Reddit is all over Google?

Because access there is arranged commercially rather than granted publicly. The file is the same for everyone, and Google’s presence is the product of an agreement. That is precisely why it is a bad foundation for someone else’s visibility plan.

Is the Claude artifact finding a bug?

No. It is how a client-rendered application behaves: the server sends an application, the browser builds the page. Google copes because it runs JavaScript. Anything that only fetches HTML sees the shell, and that is true of many crawlers, not just AI ones.

What should I check on a host before publishing there?

Three things, all free. Fetch its robots.txt and find the group for the crawler you want. Fetch the published page with a plain client and confirm your text is in the response. And ask whether the host can change either of those without telling you, because it can.

Method and sources

Every host, the path judged, the byte count of the file, the crawlers disallowed there and the user-agent group that decided each verdict are in our measurement register, along with the artifact shell figures. The register also holds the two hosts that produced no verdict, in their own bucket.

The crawler names come from the operators’ own documentation: OpenAI’s crawler list, Google’s crawler list, and Anthropic’s. The group-matching rules are in Google’s robots.txt specification.

If you want to repeat this against your own list of hosts, the whole method is one request per host and the parsing rules above. It will take you longer to choose the hosts than to run it.