Before You Ask Why Perplexity Never Cites You, Check Whether It Can Read You

If a Cloudflare zone has Bot Fight Mode switched on, PerplexityBot is being challenged and most of its requests never reach a page. Googlebot, bingbot, GPTBot, ClaudeBot and Applebot are not affected, because they are on Cloudflare’s verified bot list and PerplexityBot is not. The test almost everyone runs to check this cannot detect it.

That last sentence is the part worth your time. Sending a crawler’s user agent with curl and getting a 200 back proves nothing at all, and it is the check that gets run.

Disclosure: EchoWi sells AI visibility measurement, so a page arguing that you should check your crawler access before buying a visibility report is a page with an interest. Everything here is checkable without us: Cloudflare’s own documentation is linked at each claim, and the query that answers it for your own domain is below.

The short version

1. Bot Fight Mode challenges any bot Cloudflare has not verified. It is the free-plan bot control, it is a single on or off switch, and it is on by default for many zones.
2. PerplexityBot is not on Cloudflare’s verified bot list. Googlebot, bingbot, GPTBot, ClaudeBot, Amazonbot and Applebot are, which is why the problem shows up as exactly one engine failing while everything else looks healthy.
3. A WAF rule cannot fix it. Cloudflare documents that Bot Fight Mode “cannot be bypassed with custom rule Skip actions” because it does not run on the Ruleset Engine.
4. An IP Access rule can. The same documentation notes it “will not trigger if an IP Access rule matches the request first”, and Perplexity publishes the addresses its crawler uses.
5. Testing with a spoofed user agent proves nothing. The decision is made on source IP and verified-bot signature, so your own machine sending PerplexityBot as its user agent will always be allowed through.

Why the usual test cannot see it

Here is the check that gets run, in some form, in every technical audit:

curl -A "PerplexityBot/1.0" https://example.com/some-article/

A 200 comes back, the auditor writes “AI crawlers can access the site”, and the row goes green.

The 200 is real and it means nothing, because you are not PerplexityBot. Cloudflare decides whether to challenge a request using the source IP and whether that IP belongs to a verified crawler, not the string in the User-Agent header. A request from your laptop carrying a crawler’s name is an ordinary request from your laptop, and it is treated like one.

So the test answers a question nobody asked: does this URL return HTML to a normal client. The question that matters, does the engine’s own infrastructure receive that HTML, is invisible from outside.

The same reasoning applies to any crawler check that runs from your own machine, including the ones in most SEO tools, because they also make the request themselves. A robots.txt that allows every AI crawler is necessary and it is not sufficient: robots.txt is a request to a well-behaved crawler, and a challenge from the CDN happens before anything reads it. Run across every tool in our catalogue, that necessity turns out to be unanimous and therefore uninformative: all sixty-six pass the three checks a free audit runs, this site included, on a domain whose CDN was challenging a crawler at the time.

What is actually happening

Cloudflare classifies bot traffic by whether the bot is on its verified bot list, a list with published requirements and a policy for removing bots that breach them. Bot Fight Mode, the free plan’s bot control, issues a computationally expensive challenge to traffic matching known bot patterns, and verified bots are the ones it leaves alone.

PerplexityBot does not appear among the currently verified crawlers. Perplexity-User, the agent that fetches a page because a person asked for it, appears only in a legacy category. Cloudflare’s policy page is explicit that bots breaching its requirements “will be removed from the global allowlist”.

The result is a failure with a very particular shape, and it is why it survives audits: one engine fails and every other engine looks perfect. A dashboard showing Googlebot, GPTBot and ClaudeBot all crawling normally reads as a healthy site, and the absence of one crawler in a list of a dozen is not something anyone notices.

Why you cannot fix it with a rule

The instinct is to write a WAF rule that skips bot protection for this crawler. Cloudflare’s documentation closes that door in as many words:

You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. This is because Bot Fight Mode does not run on the Ruleset Engine, where Skip, Bypass, and Allow actions have no effect.

So on a free plan there are three real options, and only one of them is free and narrow:

The third works because of a sentence in the same documentation: Bot Fight Mode “will not trigger if an IP Access rule matches the request first”. An IP Access rule in allow mode is evaluated before it, so a request from a listed address is never challenged.

One detail will trip you up. Perplexity publishes its ranges as CIDR blocks, and Cloudflare’s IP Access rules accept only /16, /24 and single addresses for IPv4. A /29 or a /30 is rejected with a validation error, so the blocks have to be expanded into individual addresses before you can enter them.

How to check your own domain in one query

Do not test this from your own machine. Read what your CDN actually did, which is the only place the decision is recorded.

If you are on Cloudflare, the analytics API answers it directly. Group the last day of requests by user agent and response status, and look at the ratio of 403s per crawler:

httpRequestsAdaptiveGroups(
  limit: 500,
  filter: { datetime_geq: $since, datetime_lt: $until },
  orderBy: [count_DESC]
) { count dimensions { userAgent edgeResponseStatus } }

Two things to watch when you read the result.

Do not filter by a list of crawler names you wrote. Aggregate every user agent and then look at what has a high block rate. A list you write in advance can only find the crawlers you already thought of, and the one that matters is the one you forgot.

Do not read an aggregate that spans a change. If you fix something and then measure a window covering both sides of the fix, you get the mean of two different regimes and a number that describes neither. Split the window at the change and read it by hour.

Without Cloudflare, the same answer is in your origin access logs: group by user agent, count non-200 responses, and compare crawlers against each other rather than against an absolute.

What this does and does not mean for your visibility

Being readable is a precondition, not a strategy. A crawler that can reach your pages will not necessarily cite them, and most of what we measure is about what gets cited once everything is reachable: four wordings of one question can share no reliably cited source at all, and the durable citation slots in a category often belong to intermediaries rather than to the organisations being asked about.

But a blocked crawler makes all of that unmeasurable in one direction. If a tool reports that an engine never cites you, there are two explanations and they look identical from the outside: the engine read you and chose something else, or the engine never read you. Only your own logs separate them, and only one of the two is worth building content to fix.

That is the practical order of operations. Check that every engine you care about receives your HTML. Then measure what they cite. Doing it the other way round means paying for a report on a question that was decided at the CDN.

Common Questions About Crawler Access

Does Cloudflare block PerplexityBot by default?

Not by default in the sense of a deliberate rule against it, but Bot Fight Mode challenges bots that are not on Cloudflare’s verified list, and PerplexityBot is not on it. Any zone with Bot Fight Mode enabled is challenging it, while Googlebot, bingbot, GPTBot, ClaudeBot, Amazonbot and Applebot pass because they are verified.

Why does testing with curl and a crawler user agent show it working?

Because the user agent is not what the decision is made on. Cloudflare evaluates the source IP and whether it belongs to a verified crawler, so a request from your own machine carrying a crawler’s name is treated as a request from your own machine. It will be allowed, and it tells you nothing about what the real crawler receives.

Can I add a WAF rule to let a specific crawler through Bot Fight Mode?

No. Cloudflare documents that Bot Fight Mode cannot be bypassed or skipped with custom rules or page rules, because it does not run on the Ruleset Engine where skip actions apply. Super Bot Fight Mode, on paid plans, does support skip rules.

What is the free way to allow one crawler without turning bot protection off?

An IP Access rule in allow mode for the crawler’s published addresses. Cloudflare’s documentation states Bot Fight Mode will not trigger if an IP Access rule matches first. Note that IPv4 access rules accept only /16, /24 and single addresses, so published CIDR blocks smaller than a /24 have to be expanded into individual IPs.

How do I know if this is affecting my site right now?

Read your CDN or origin logs, not a synthetic request. Group the last day by user agent and response status and compare the block rate across crawlers. The signature is unmistakable: one crawler with a high proportion of 403s while every other crawler sits near zero.

Does allowing a crawler’s IPs weaken my security?

It opens exactly the addresses you name and nothing else. The alternative on a free plan is switching bot protection off for everything, which is a far larger change. What you should not do is allow whatever IP claims to be a given crawler in its user agent, because that string can be sent by anyone and allowing on that basis has no security value.

Where this leaves you

The reason this defect survives is that it fails quietly and it fails in one place. Nothing errors, no page breaks, the site is fast, robots.txt is permissive, and eleven crawlers out of twelve are perfectly happy. The twelfth is simply absent, and absence looks the same as indifference.

So run the check on your logs before you commission any report about who cites you. It takes one query, it costs nothing, and it is the only version of this question that has a real answer.