Which AI crawlers can read your website?
Enter a website address. We read its robots.txt file and tell you, crawler by crawler, whether it may visit: for training, for AI search, or when a person asks an assistant about you.
The crawlers we check
Each name below is what a company’s crawler calls itself in robots.txt. What each one does is taken from that company’s own page.
| Name in robots.txt | Used for | What it does | Source |
|---|---|---|---|
| OpenAI | |||
GPTBot | Training | Collects pages that may be used to train OpenAI’s models. | OpenAI’s page |
OAI-SearchBot | AI search | Builds the index that ChatGPT search uses to find and link to websites. | OpenAI’s page |
ChatGPT-User | User request | Fetches a page when a ChatGPT user’s request calls for it. OpenAI says robots.txt rules may not apply to these visits.OpenAI says this one may fetch a page anyway when a user asks for it. | OpenAI’s page |
| Anthropic | |||
ClaudeBot | Training | Collects pages that may be used to train Anthropic’s models. | Anthropic’s page |
Claude-SearchBot | AI search | Indexes pages to improve the search results Claude shows. | Anthropic’s page |
Claude-User | User request | Fetches a page when a Claude user’s question calls for it. | Anthropic’s page |
| Perplexity | |||
PerplexityBot | AI search | Indexes pages so that Perplexity can show and link to them. Perplexity says it is not used to train models. | Perplexity’s page |
Perplexity-User | User request | Fetches a page when a Perplexity user’s question calls for it. Perplexity says it generally ignores robots.txt.Perplexity says this one may fetch a page anyway when a user asks for it. | Perplexity’s page |
Googlebot | AI search | Crawls for Google Search and all of its features, including AI Overviews. | Google’s page |
Google-ExtendedA name, not a crawler | Training | Not a crawler: a name that tells Google whether pages it has crawled may be used to train Gemini models and to ground Gemini’s answers. It does not affect Google Search. | Google’s page |
| Microsoft | |||
Bingbot | AI search | Crawls for Bing search, which Microsoft Copilot draws on. | Microsoft’s page |
| Apple | |||
Applebot | AI search | Crawls for search in Siri, Spotlight and Safari. If your file has no Applebot rules, Apple says it follows your Googlebot rules. | Apple’s page |
Applebot-ExtendedA name, not a crawler | Training | Not a crawler: a name that tells Apple whether pages Applebot has crawled may be used to train Apple’s AI models. | Apple’s page |
| Meta | |||
Meta-ExternalAgent | Training | Crawls for uses such as training Meta’s AI models and indexing content for its products. | Meta’s page |
Meta-WebIndexer | AI search | Indexes pages so that Meta AI can cite and link to them. | Meta’s page |
Meta-ExternalFetcher | User request | Fetches a link when a user of a Meta product asks for it. Meta says it may bypass robots.txt.Meta says this one may fetch a page anyway when a user asks for it. | Meta’s page |
| Amazon | |||
Amazonbot | Training | Crawls to improve Amazon’s products and may be used to train Amazon’s AI models. | Amazon’s page |
Amzn-SearchBot | AI search | Indexes pages for search in Amazon products such as Alexa. Amazon says it is not used to train models.Amazon says that when your file does not name this crawler, it follows the rules you give other search crawlers. This tool shows the standard rule instead. | Amazon’s page |
Amzn-User | User request | Fetches live information when an Alexa user’s question calls for it. Amazon says it may not follow all robots.txt rules.Amazon says this one may fetch a page anyway when a user asks for it. | Amazon’s page |
| Mistral AI | |||
MistralAI-Training | Training | Collects pages to build the datasets that train Mistral’s AI models. | Mistral AI’s page |
MistralAI-Index | AI search | Indexes pages for the search that Mistral’s assistant uses. Mistral says it is not used to train models. | Mistral AI’s page |
MistralAI-User | User request | Fetches a page when a user of Mistral’s assistant asks about it. | Mistral AI’s page |
| DuckDuckGo | |||
DuckAssistBot | AI search | Crawls pages in real time for DuckDuckGo’s AI-assisted answers. DuckDuckGo says this data is not used to train AI models. | DuckDuckGo’s page |
| Common Crawl | |||
CCBot | Other | Builds Common Crawl’s free public copy of the web. Common Crawl says its data is one of the most widely used sources of training data for large language models. | Common Crawl’s page |
List reviewed October 11, 2026.
Rules to copy
Pick what you want, then add the lines to the robots.txt file at the top level of your website.
Let AI search and assistants read my site, block training
Add these lines. They block the crawlers that collect pages for training. Nothing else changes.
# AI crawler rules. Crawler list reviewed 2026-10-11.User-agent: GPTBotDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: MistralAI-TrainingDisallow: /
If your file already has a section for one of these names, replace that section instead of adding a second one.
These stay allowed as long as your file does not block them: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, Meta-WebIndexer, Meta-ExternalFetcher, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User, DuckAssistBot.
Not included, because blocking them does more than stop training: Google-Extended, Meta-ExternalAgent, Amazonbot, CCBot. See the list above for what each one does.
Only if your file blocks all crawlers
A section that names a crawler replaces your rules for all crawlers for that crawler. Copy into it any Disallow lines that should still apply.
# Only for a file that blocks all crawlers with "User-agent: *" and "Disallow: /".User-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: Claude-SearchBotAllow: / User-agent: Claude-UserAllow: / User-agent: PerplexityBotAllow: / User-agent: Perplexity-UserAllow: / User-agent: Meta-WebIndexerAllow: / User-agent: Meta-ExternalFetcherAllow: / User-agent: Amzn-SearchBotAllow: / User-agent: Amzn-UserAllow: / User-agent: MistralAI-IndexAllow: / User-agent: MistralAI-UserAllow: / User-agent: DuckAssistBotAllow: /
These lines change who may read your site. They don’t change what an AI assistant says about you.
Allow everything
You don’t need to add anything: a crawler your file does not name is allowed. Remove the sections that block the crawlers you want.
Nothing to add.
Only if your file blocks all crawlers
A section that names a crawler replaces your rules for all crawlers for that crawler. Copy into it any Disallow lines that should still apply.
# Only for a file that blocks all crawlers with "User-agent: *" and "Disallow: /".User-agent: GPTBotAllow: / User-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: ClaudeBotAllow: / User-agent: Claude-SearchBotAllow: / User-agent: Claude-UserAllow: / User-agent: PerplexityBotAllow: / User-agent: Perplexity-UserAllow: / User-agent: Google-ExtendedAllow: / User-agent: Applebot-ExtendedAllow: / User-agent: Meta-ExternalAgentAllow: / User-agent: Meta-WebIndexerAllow: / User-agent: Meta-ExternalFetcherAllow: / User-agent: AmazonbotAllow: / User-agent: Amzn-SearchBotAllow: / User-agent: Amzn-UserAllow: / User-agent: MistralAI-TrainingAllow: / User-agent: MistralAI-IndexAllow: / User-agent: MistralAI-UserAllow: / User-agent: DuckAssistBotAllow: / User-agent: CCBotAllow: /
These lines change who may read your site. They don’t change what an AI assistant says about you.
Block AI crawlers
Add these lines. They ask every AI crawler on our list to stay out. Ordinary search engines are not included.
# AI crawler rules. Crawler list reviewed 2026-10-11.User-agent: GPTBotDisallow: / User-agent: OAI-SearchBotDisallow: / User-agent: ChatGPT-UserDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Claude-SearchBotDisallow: / User-agent: Claude-UserDisallow: / User-agent: PerplexityBotDisallow: / User-agent: Perplexity-UserDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: Meta-ExternalAgentDisallow: / User-agent: Meta-WebIndexerDisallow: / User-agent: Meta-ExternalFetcherDisallow: / User-agent: AmazonbotDisallow: / User-agent: Amzn-SearchBotDisallow: / User-agent: Amzn-UserDisallow: / User-agent: MistralAI-TrainingDisallow: / User-agent: MistralAI-IndexDisallow: / User-agent: MistralAI-UserDisallow: / User-agent: DuckAssistBotDisallow: / User-agent: CCBotDisallow: /
If your file already has a section for one of these names, replace that section instead of adding a second one.
Not included: Googlebot, Bingbot, Applebot. Blocking them removes your site from ordinary search.
ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User: their makers say these may fetch a page anyway when a user asks for it. robots.txt can’t stop that; a firewall rule can.
These lines change who may read your site. They don’t change what an AI assistant says about you.
What this tool can’t see
- Blocks made by a firewall, a CDN or your host. A crawler can be allowed in robots.txt and still be turned away.
- Rules inside your pages: “noindex”, “nosnippet” and similar tags and headers.
- Whether a crawler follows your rules. robots.txt is a request. The companies on our list say which of their crawlers honour it; we repeat only what they say.
- Whether a crawler has visited you. That is in your server logs.
- Other websites. Each host has its own file: example.com, www.example.com and shop.example.com are checked separately.
- Crawlers we don’t list. The list is checked against each company’s own page; the date is under the table.
Questions
See what else AI can’t read on your site
robots.txt is one of six things the free check looks at. It also asks an AI model your customers’ most common questions.
Run the free check