> ## Documentation Index > Fetch the complete documentation index at: https://snowseo.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Crawler registry > Every AI crawler beacon recognises, what it is for, and why the list is shorter than you might expect. 54 agents from 21 providers, matched on documented tokens only. The registry lives in [`crawlers.ts`](https://github.com/Snow-SEO/beacon/blob/main/packages/beacon/src/crawlers.ts) and classification happens on the server, so this list improves without you upgrading anything. **Missing one?** [Open an issue](https://github.com/Snow-SEO/beacon/issues/new?template=crawler_registry.yml) with the token and a link to the provider's own documentation of it. *** ## Why the list is not longer **Tokens, not vendor names.** Matching the substring `claude` or `openai` would catch a researcher's curl script and any User-Agent that merely mentions the vendor. Every entry here is a token the operator published. **Googlebot and Bingbot are excluded on purpose.** Both are far higher volume than any AI crawler and are already covered by Search Console and Bing Webmaster Tools. Including them would drown the numbers you came for. `Google-Extended` and `GoogleOther` are the AI-specific ones, and they are here. **A wrong entry is worse than a missing one.** An agent added on the strength of a log line silently mislabels real traffic, and nobody notices. That is why a documentation link is required. *** ## What is matched without a token Some clients identify as nothing useful. Claude Code arrives as `axios/1.8.4`. A request that **negotiates for Markdown** is counted as an AI client even with no registry match, and recorded as its bare product token - `axios`, `python-requests` - with the provider resolved from the IP address rather than the User-Agent, since that is the direction that cannot be faked. A browser opening a `.md` URL trips the same signal, so requests carrying a full `Sec-Fetch-*` header set are vetoed. Unless they came from a datacenter range: agents now drive real headless engines, and a real DeepSeek fetcher has arrived as Firefox, with the complete Sec-Fetch set, from Huawei Cloud. The address is the part it could not dress up. *** ## The categories ### Answer fetch `answer_fetch` - 16 agents. A person asked their assistant something and it is reading your page to answer them, right now. The highest-value category: these hits map to a real reader. | Provider | Agent | Token matched | | ----------- | --------------------- | ----------------------- | | OpenAI | ChatGPT-User | `chatgpt-user` | | Anthropic | Claude-User | `claude-user` | | Perplexity | Perplexity-User | `perplexity-user` | | Google | Gemini-Deep-Research | `gemini-deep-research` | | | Google-GeminiNotebook | `google-gemininotebook` | | | Google-NotebookLM | `google-notebooklm` | | | Google-Read-Aloud | `google-read-aloud` | | | Google-Agent | `google-agent` | | Microsoft | Copilot | `copilot` | | xAI | Grok-DeepSearch | `grok-deepsearch` | | Mistral | MistralAI-User | `mistralai-user` | | Amazon | Amzn-User | `amzn-user` | | Meta | Meta-ExternalFetcher | `meta-externalfetcher` | | DuckDuckGo | DuckAssistBot | `duckassistbot` | | Moonshot AI | Kimi-User | `kimi-user` | | Alibaba | Qwen-User | `qwen-user` | ### Search index `search_index` - 10 agents. Building an index behind an AI search product. Closer to a traditional crawler, but the index feeds answers rather than a results page. | Provider | Agent | Token matched | | ----------- | ---------------- | ------------------ | | OpenAI | OAI-SearchBot | `oai-searchbot` | | Anthropic | Claude-SearchBot | `claude-searchbot` | | Perplexity | PerplexityBot | `perplexitybot` | | xAI | xAI-SearchBot | `xai-searchbot` | | Mistral | MistralAI-Index | `mistralai-index` | | Apple | Applebot | `applebot` | | Amazon | Amzn-SearchBot | `amzn-searchbot` | | Meta | Meta-WebIndexer | `meta-webindexer` | | Moonshot AI | Kimi-SearchBot | `kimi-searchbot` | | ByteDance | TikTokSpider | `tiktokspider` | ### Training `training` - 13 agents. Collecting corpus for model training. High volume, no immediate reader. This is the category most people want to block. | Provider | Agent | Token matched | | ------------ | ---------------------------- | ------------------------------ | | OpenAI | GPTBot | `gptbot` | | Anthropic | ClaudeBot | `claudebot` | | | anthropic-ai | `anthropic-ai` | | Google | Google-CloudVertexBot | `google-cloudvertexbot` | | | Google-Extended | `google-extended` | | Apple | Applebot-Extended | `applebot-extended` | | Amazon | Amazonbot | `amazonbot` | | Meta | Meta-ExternalAgent | `meta-externalagent` | | ByteDance | Bytespider | `bytespider` | | Cohere | cohere-training-data-crawler | `cohere-training-data-crawler` | | | cohere-ai | `cohere-ai` | | Allen AI | AI2Bot | `ai2bot` | | Common Crawl | CCBot | `ccbot` | ### AI crawler `ai_crawler` - 15 agents. Something else a model vendor runs, where the purpose is not published clearly enough to place it above. | Provider | Agent | Token matched | | ----------- | --------------- | ----------------- | | OpenAI | OAI-AdsBot | `oai-adsbot` | | Google | GoogleOther | `googleother` | | xAI | xAI-Web-Crawler | `xai-web-crawler` | | | GrokBot | `grokbot` | | | xAI-Bot | `xai-bot` | | Moonshot AI | KimiBot | `kimibot` | | Alibaba | QwenBot | `qwenbot` | | | TongyiBot | `tongyibot` | | | AliyunBot | `aliyunbot` | | ByteDance | Doubaobot | `doubaobot` | | Baidu | ERNIEBot | `erniebot` | | | YiyanBot | `yiyanbot` | | DeepSeek | DeepSeekBot | `deepseekbot` | | Zhipu AI | ChatGLM-Spider | `chatglm-spider` | | You.com | YouBot | `youbot` | *** ## Verification Being in this list only means the token was recognised. Whether the request really came from that provider is a separate question, answered by [three independent signals](/docs/beacon/overview#how-a-crawler-gets-verified) and recorded as one of four states. Providers that publish IP ranges or an authenticating reverse-DNS suffix can be verified. Those that publish neither stay `unverified`, which means "nothing to check against" and not "suspicious".