> ## Documentation Index
> Fetch the complete documentation index at: https://snowseo.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Why AI assistants cannot read your site, and why crawler counts look low.

Two different problems produce the same symptom - an empty Bot Traffic dashboard - and they have nothing to do with each other.

| Symptom                                    | Cause                                                    | Where to look                                          |
| ------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------ |
| ChatGPT or Claude says it got a 403        | something refuses AI crawlers before your site sees them | the reachability panel, below                          |
| Counts look far lower than your server log | a page cache answers without running your site's code    | [page caching](/docs/beacon/install/wordpress#page-caching) |

***

## AI assistants are blocked

The panel at the top of **Bot Traffic** fetches your site from SnowSEO with every published crawler user agent - ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User, PerplexityBot - and compares each response with what a browser gets from the same place at the same moment. A difference is the finding, and the response headers usually say which layer produced it.

This runs from outside because it has to. A block at Cloudflare or at your host happens above your site entirely; from the inside there is nothing to see.

### Cloudflare

The most common cause, and the one with the most moving parts.

* **Security → Bots** carries Search, Agent and Training categories. From **15 September 2026** Cloudflare blocks Training and Agent by default on ad-bearing pages for new zones *and existing free zones*, unless you opt out.
* **Bot Fight Mode** (free plan) issues CPU challenges that no crawler can solve. It runs outside the Ruleset Engine, so a WAF Skip rule cannot bypass it - the only fix is turning it off, or moving to Super Bot Fight Mode on Pro.
* **AI Crawl Control / pay-per-crawl** answers `402` at the edge. The request never reaches your origin.
* A **WAF custom rule** matching `http.user_agent` will do it too. On Pro and above, add a Skip rule.

<Note>
  Multi-purpose crawlers such as Googlebot and Applebot are classified as both Search *and* Training, so a restrictive Training policy can take out search crawling with it.
</Note>

### Managed hosting

| Host                                      | Behaviour                                                                                     | Fix                                                                                 |
| ----------------------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **WP Engine**                             | Rate-limits ClaudeBot, GPTBot, Amazonbot and Bytespider at the platform level. Returns `429`. | Support ticket. This cannot be disabled per-bot from the customer Web Rules Engine. |
| **SiteGround**                            | Blocks training crawlers server-side by default; allows user-triggered ones.                  | Support.                                                                            |
| **Kinsta, Pantheon, Pressable, Flywheel** | No platform-level block.                                                                      | Nothing to do.                                                                      |

### robots.txt

Each token is independent. Blocking `ClaudeBot` does **not** block `Claude-User`, and blocking `GPTBot` does not block `ChatGPT-User`. That asymmetry is why a site can believe it is open to assistants while blocking the exact agent a person's question would use, or the reverse.

The reachability panel evaluates every token against your live `robots.txt` and names the one that matched.

### Security plugins and server WAFs

* **Imunify360** (common on cPanel shared hosting) ships ModSecurity rules blocking 16 AI bots - rule IDs `88345386`-`88345402`. Ask your host to disable them.
* **Wordfence** has no deliberate AI blocking, but its rate limiting and bad-bot heuristics throttle crawlers. There is no "allow well-known AI crawlers" toggle.
* **Blackhole for Bad Bots** bans anything that follows its honeypot link, and AI crawlers are not on its default whitelist. Add them and clear the existing ban.
* **OWASP CRS** blocks AI training bots by default while allowing Google and Bing.

Anything blocked at the edge leaves no trace in a plugin's own log, because the plugin never ran.

***

## Checking by hand

```bash theme={null}
# Baseline
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" https://example.com/

# Live assistant fetch - the one that matters most
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com/
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)" https://example.com/
```

Anything other than a matching `200` is a finding. Watch for `429` (host rate limit) and `307` (redirect to a challenge) as well as `403`.

Run this from outside your own network. A request from an address your firewall trusts proves nothing about one from Anthropic's or OpenAI's.

***

## What is genuinely unmeasurable

Not every gap is a misconfiguration.

* **Google AI Overviews and AI Mode** do not fetch your page. They answer from the Search index that Googlebot already built, so there is no request to count and no user agent to see.
* **Answers from training data** involve no request at all.
* **Cached assistant fetches.** Anthropic's `web_fetch` caches by default, so a repeated fetch of the same URL may never reach you.

These are limits of the medium, not of the integration.