How robots.txt works
robots.txt is a text file at the root of your site. It groups rules by crawler name, called the user agent. The format is an internet standard, RFC 9309. Following it is voluntary. Well-behaved crawlers do. The companies below say their training and search crawlers follow it. OpenAI and Perplexity say the fetchers that act on a user’s request may not.
Three kinds of AI crawler
- Training crawlers collect pages that may be used to train models.
- Search crawlers build the index an assistant searches when it answers.
- User-triggered fetchers load one page because a person asked the assistant about it.
The difference matters. Blocking a training crawler does not remove you from an assistant’s search. Blocking a search crawler can.
The crawlers, as each company documents them
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetches for a user’s request).
- Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (fetches for a user’s request).
- Perplexity: PerplexityBot (its search index), Perplexity-User (fetches for a user’s question).
- Google: Googlebot crawls for Google Search, which AI Overviews draw on. Google-Extended is a robots.txt name, not a separate crawler. It controls whether your content is used for Gemini, and it does not affect Search.
- Microsoft: Bingbot crawls for Bing, which Copilot draws on.
- Apple: Applebot crawls for Apple’s search features. Applebot-Extended is a robots.txt name that controls use of your content for Apple’s AI training.
Names and rules change. Before you edit your file, read each company’s own crawler page.
Letting them in
Most business websites want customers to find them, so most should let these crawlers in. If your file has no rules for a crawler by name, the rules under “User-agent: *” apply. If there are none, the crawler is allowed.
User-agent: *Allow: /Blocking training but not search
User-agent: GPTBotDisallow: / User-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: /Common mistakes
- A leftover “Disallow: /” for every crawler, from when the site was being built.
- A firewall or bot-protection service that blocks AI crawlers even though robots.txt allows them.
- Blocking a search crawler when you meant to block training.
- Assuming a block removes what a model has already learned. It doesn’t.
- A robots.txt that answers with a server error. The standard tells crawlers to treat that as “keep out”.
Where FactHound helps, and where it doesn’t
What FactHound does
- The free check reads your robots.txt and tells you which of ten AI crawler names are blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Bingbot and Applebot-Extended.
- It also tells you when your website blocks the check itself, which often means a firewall is blocking crawlers too.
- AI crawler checker
What it doesn’t do
- FactHound doesn’t edit your robots.txt or your firewall.
- The free check tests the rules for your homepage, not for every page.
- It can’t tell you whether a crawler has visited.