crawlers.ts and classification happens on the server, so this list improves without you upgrading anything.
Missing one? Open an issue with the token and a link to the provider’s own documentation of it.
Why the list is not longer
Tokens, not vendor names. Matching the substringclaude or openai would catch a researcher’s curl script and any User-Agent that merely mentions the vendor. Every entry here is a token the operator published.
Googlebot and Bingbot are excluded on purpose. Both are far higher volume than any AI crawler and are already covered by Search Console and Bing Webmaster Tools. Including them would drown the numbers you came for. Google-Extended and GoogleOther are the AI-specific ones, and they are here.
A wrong entry is worse than a missing one. An agent added on the strength of a log line silently mislabels real traffic, and nobody notices. That is why a documentation link is required.
What is matched without a token
Some clients identify as nothing useful. Claude Code arrives asaxios/1.8.4.
A request that negotiates for Markdown is counted as an AI client even with no registry match, and recorded as its bare product token - axios, python-requests - with the provider resolved from the IP address rather than the User-Agent, since that is the direction that cannot be faked.
A browser opening a .md URL trips the same signal, so requests carrying a full Sec-Fetch-* header set are vetoed. Unless they came from a datacenter range: agents now drive real headless engines, and a real DeepSeek fetcher has arrived as Firefox, with the complete Sec-Fetch set, from Huawei Cloud. The address is the part it could not dress up.
The categories
Answer fetch
answer_fetch - 16 agents. A person asked their assistant something and it is reading your page to answer them, right now. The highest-value category: these hits map to a real reader.
Search index
search_index - 10 agents. Building an index behind an AI search product. Closer to a traditional crawler, but the index feeds answers rather than a results page.
Training
training - 13 agents. Collecting corpus for model training. High volume, no immediate reader. This is the category most people want to block.
AI crawler
ai_crawler - 15 agents. Something else a model vendor runs, where the purpose is not published clearly enough to place it above.
Verification
Being in this list only means the token was recognised. Whether the request really came from that provider is a separate question, answered by three independent signals and recorded as one of four states. Providers that publish IP ranges or an authenticating reverse-DNS suffix can be verified. Those that publish neither stayunverified, which means “nothing to check against” and not “suspicious”.