The New York Times and CNBC Block Every AI Crawler We Checked. Google's AI Overview Cites Them Both Anyway.

Every study of AI citation, ours included, records which domains an engine cites and stops there. Nobody checks whether the citation resolves, which costs one HTTP request. We took every URL printed in nine AI Overview answers across four languages, 79 citations over 58 hosts, and fetched all of them. None was dead. Then we read each cited host’s robots.txt, and found that 11 of them disallow AI crawlers outright while being cited in the answer. Two of those, The New York Times and CNBC, disallow all ten crawlers we checked including Google-Extended, and each is the second source in its answer.

The useful part is not the contradiction. It is that the thing people are told to do about AI crawling, and the thing that decides whether you appear in an AI answer, are two different systems that do not talk to each other.

Disclosure: EchoWi sells AI visibility measurement, so a finding that robots.txt does not control AI Overview presence is one we benefit from publishing. The nine questions, the market and language of each, every URL with its HTTP status and every robots.txt line quoted below are in our measurement register, and the robots.txt files are public, so this can be repeated against us in an afternoon.

The short version

1. 79 citations, 58 hosts, 9 questions, 4 languages, one AI Overview draw each. Four in the United States in English, two in Spain, two in Germany and one in France, each in its own language.
2. Nothing was dead. 71 URLs answered, 8 refused a plain HTTP client with 403, and none returned 404.
3. Nothing had moved either. Zero of the 79 redirected anywhere other than themselves, so the URL an answer prints is the URL that serves the page.
4. Of the 55 hosts that let us read robots.txt, 17 name an AI crawler at all. The other 38 have simply never written a rule about one.
5. 11 of those 17 disallow at least one crawler outright, and all 11 are cited anyway.
6. The New York Times and CNBC each disallow all ten we checked, Google-Extended included. Wirecutter is citation two for the headphones question and CNBC is citation two for payroll.
7. Forbes disallows six and explicitly allows OAI-SearchBot, and is citation one for the same question. That is not an oversight, it is a policy: block the training crawler, admit the search crawler. That policy is not unique to Forbes: reading the same files across the hosts people are told to publish on, training crawlers are refused far more often than search crawlers, and eleven of those hosts admit Googlebot while shutting at least one AI crawler out.
8. So a robots.txt tells you nothing about whether a site will appear in AI Overview, in either direction, which also means a competitor’s robots.txt tells you nothing about theirs.

What we did

Nine buying questions, each asked once of Google’s AI Overview, in the market and language a buyer would ask them in. Four are business software, where our register already has rows to compare against. The rest are consumer categories, travel credit cards and headphones, added in two later passes: first because the publishers that write AI crawler rules are news publishers and business software questions barely cite any, and then in Spain, France and Germany because every blocker the first passes found was American. One draw per question is deliberate and it is the reason nothing here is expressed as a rate: the unit is the URL an answer printed, not how often it prints it. Our register holds separate studies for how often a source repeats, such as what a durable source count actually measures, and those are the ones to quote for that.

For each citation we made one GET request, followed redirects, and recorded the final status and the final URL.

GET rather than HEAD, and that choice has a scar behind it. Our own link checker used HEAD when it was first written and reported five dead sources; two of the five answered 200 to a GET. A method a server is not obliged to support is not evidence that a page is missing, and with the HEAD list we would have annotated two perfectly live citations as broken.

Then, for every distinct host, we fetched https://host/robots.txt and looked for a group addressed to each of ten named AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot, CCBot, Applebot-Extended, Bytespider and anthropic-ai. A host counts as blocking a crawler only when the group addressed to that agent disallows everything.

Nothing was dead, and that is worth one paragraph

71 of 79 answered. 8 returned 403 to the fetcher we used. Zero returned 404, 410 or anything else in the dead range, and zero redirected to a different address.

We expected to find at least a few stale citations, because that is the shape of the complaint people make about generated answers. In this sample there are none. The pages an answer points at are there.

The 403s are not a finding about those pages and we are not going to dress them up as one. They are bot protection refusing an unfamiliar client from one location, and the engine that cited them obviously got in. That distinction is not a technicality, it is the same one that makes the rest of this piece work: what a server does to you is not what it does to a crawler.

The publishers who block a crawler and get cited anyway

The two lines that carry the argument were read back by hand rather than trusted to the parser that found them, because a parser that defines its own universe is the most common way a study of this kind goes wrong:

www.cnbc.com/robots.txt
User-agent: Google-Extended
Disallow: /

www.forbes.com/robots.txt
User-agent: GPTBot
Disallow: /

User-Agent: OAI-SearchBot
Allow: /

CNBC is the second citation in the AI Overview answer for “What is the best payroll service for a small business?”. Forbes is the first.

The ones who never wrote a rule

The 11 blockers are the interesting minority. The larger number in the table is the one underneath: of the 55 hosts whose robots.txt we could read, 38 do not name a single one of the ten AI crawlers. Not to allow them, not to block them. They have no position.

That is worth holding next to what the category sells. A whole industry of advice has grown up around AI crawler directives, and most of the publishers actually holding citation slots in these answers have never written one. They are cited because Googlebot indexed them, which was already true before any of this existed.

It also sets the price of the advice correctly. If you write a rule, you are joining a minority of 17 in this sample, and 11 of those 17 are being cited in spite of what they wrote. The rule is not what is holding the slot. Our separate audit of what this category itself implements found the same ordering, with vendors adopting the convention that has no demonstrated reader and skipping the one with a specification.

Why this happens, and why it is not a loophole

Google-Extended is not the crawler that fetches pages for Google. It is a control over whether content Google already has may be used for Gemini and for grounding. AI Overview is assembled from Google’s search index, which Googlebot builds, and no publisher in this sample disallows Googlebot, because that would remove them from search altogether.

So the sequence is: Googlebot crawls, the index is built, AI Overview draws its answer from that index, and the rules a publisher wrote for GPTBot and Google-Extended were never in that path.

That is not a trick anyone is playing. Forbes has written the most coherent configuration in the table: keep out the crawler that builds a training corpus, admit the crawler that answers a search. What it demonstrates is that these are separate levers, and only one of them is connected to the thing most people are worried about losing.

And it is not an American habit

Every blocker the first two passes found was American: CNBC, Forbes, The New York Times, NerdWallet. That is a real confound and the piece could not tell “publishers block” from “American publishers block”, because the Spanish and German questions in it were business software, which barely cites a news publisher at all.

So we asked three more, one per European market, in the categories that had produced the American blockers: travel credit cards in Spain and Germany, headphones in France.

They block too. Viajes National Geographic in Spain disallows seven of the ten, GPTBot among them, and is the first citation in that answer. 01net disallows two and Frandroid one, both French technology titles, both cited. Blockers now appear in all four markets.

The tidy version of that does not survive. Reading the European rows alone, the pattern looks like Americans blocking the major crawlers and Europeans blocking only the low-value scrapers: the two French titles disallow CCBot and Bytespider and let GPTBot through. It fits those rows perfectly. It dies against the rest of the table, because Spain’s National Geographic blocks GPTBot and the United States’ own Equal Pay Today blocks nothing but CCBot. Eleven blockers across four markets cannot support a claim about market temperament, and we are not going to make one.

The one contrast worth naming is a single publisher rather than a country. BILD explicitly allows PerplexityBot, CCBot, Applebot-Extended and Bytespider, which is the opposite posture to the other cited news publishers here and is what a licensing arrangement looks like written into a robots.txt.

What this changes for three different people

If you publish and you have blocked AI crawlers to protect your work, you have not removed yourself from Google’s AI answers, and you may not have intended to. It is worth knowing which of your rules is doing what, because the visible outcome, being quoted without a click, is produced by the crawler you cannot block without leaving search.

If you sell or buy AI visibility, a competitor’s robots.txt is not intelligence about their presence. Four of the eight hosts here that mention an AI crawler at all are blocking one and being cited regardless. Any audit that reads robots.txt and infers who will show up in an answer is reading the wrong file.

If you build measurement tools, the 403s matter more than the blocks. 8 of 79 cited URLs refuse a plain client. A checker that runs from one address and marks those unreachable would report those 8 sources of a live answer as broken, and be wrong about every one of them.

What this does not show

Nine questions, one surface, one draw each, one day. This is Google AI Overview on 14 August 2026 in four markets. It says nothing about ChatGPT, Perplexity or Gemini, which retrieve differently and would each need their own measurement.

The categories are not a sample. They were chosen because our register already covers them, so these rows can be compared against earlier ones. That is the opposite of a random draw.

Seventeen hosts naming a crawler is still a small number to reason from, and 11 of 17 is not a rate we would defend as the industry’s. What the 11 support is an existence claim, which is all the argument needs: blocking AI crawlers is compatible with being a top source in an AI answer, and here are 11 sites doing it. This piece has now been widened twice, from 4 of 8 to 8 of 12 to 11 of 17, and each time the proportion held rather than collapsing. That is worth saying because it could have gone the other way, and a reader cannot tell from the final number that it did not.

Three hosts would not serve us robots.txt at all, answering 403, and they are counted apart. They are counted separately from hosts that permit everyone, because a challenge is not a permission, and we do not know what they declare.

Nothing here is causal. We did not change a robots.txt and watch a citation move. That experiment is possible and nobody has published it.

Common Questions About Blocking AI Crawlers

Does blocking GPTBot remove me from Google’s AI Overview?

No. GPTBot is OpenAI’s crawler and has no part in how Google builds AI Overview, which draws on Google’s search index. In our sample, 11 publishers disallow AI crawlers and all 11 are cited in AI Overview answers, including two that disallow all ten crawlers we checked.

Does Google-Extended control whether I appear in AI Overview?

Not in anything we observed. Google-Extended governs whether content may be used for Gemini and for grounding, not whether Googlebot crawls you or whether the search index feeds an AI Overview. CNBC disallows Google-Extended and is the second cited source in the payroll answer we measured on 14 August 2026.

Do AI answers cite pages that no longer exist?

Not in this sample. We fetched all 79 URLs that nine AI Overview answers printed and none returned a dead status, and none redirected elsewhere. Eight returned 403 to the fetcher we used, which is bot protection rather than a missing page.

Can I tell from a competitor’s robots.txt whether they will show up in AI answers?

No, and this is the practical consequence. Of the 17 cited hosts that name an AI crawler at all, 11 block one and are cited anyway. A robots.txt describes what a site asks named crawlers to do, not what surface will quote it.

Should I check cited sources with a HEAD request?

No. A server is not obliged to support HEAD, and treating a HEAD failure as a missing page produces false positives. Our own link checker did exactly that and reported two live pages as dead before it was changed to GET with redirects followed.