Which AI crawlers should you allow?
Allow the crawlers that put your pages in AI search answers, and decide separately about the ones that collect content for model training. For most UK businesses that want to be found, that means allowing the search and answer bots and, if you wish, opting out of training. Each vendor now publishes separate tokens for these jobs, so you no longer have to choose all or nothing.
What are the three kinds of AI crawler?
Vendor documentation describes three roles, and mixing them up is the most common robots.txt mistake.
- Training crawlers collect public content that may be used to train models. Blocking them does not remove you from search.
- Search or answer crawlers index pages so they can be cited and linked in AI search results. Blocking them can keep you out of those answers.
- User-triggered fetchers visit a page because a person asked an assistant about it. Several vendors say robots.txt may not apply to these.
Which crawlers do the vendors document?
The table below only includes crawlers whose official documentation we could find and read. Always check the vendor page before relying on a token, because names and policies change.
| Crawler (token) | Company | Purpose | Respects robots.txt per docs |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes, disallowing it signals content should not be used for training |
| OAI-SearchBot | OpenAI | ChatGPT search features | Yes, blocking keeps you out of ChatGPT search answers |
| ChatGPT-User | OpenAI | User-triggered fetch | Docs say rules may not apply |
| ClaudeBot | Anthropic | Training | Yes, Anthropic says its bots honour robots.txt |
| Claude-SearchBot | Anthropic | Improves search result relevance | Yes, per the same article |
| Claude-User | Anthropic | User-triggered fetch | Yes, per the same article |
| Googlebot | Search, Discover, Images, News | Yes | |
| Google-Extended | Gemini training and grounding (control token only) | Yes, set as a robots.txt group | |
| PerplexityBot | Perplexity | Search results, not model training | Yes |
| Perplexity-User | Perplexity | User-triggered fetch | Generally ignores it |
| Applebot | Apple | Spotlight, Siri, Safari search; may also help train Apple models | Yes |
| Applebot-Extended | Apple | Opt-out of generative AI training (control token only) | Yes |
| CCBot | Common Crawl | Open web crawl repository | Yes, via a CCBot group |
| meta-externalagent | Meta | Training or indexing for products | Block via a Disallow rule |
| meta-webindexer | Meta | Improves Meta AI search quality | Block via a Disallow rule |
| meta-externalfetcher | Meta | User-triggered fetch | Docs say it may bypass robots.txt |
We left Bingbot out of the table because we could not load Microsoft's own crawler documentation. Bing is an ordinary search crawler, so check its Webmaster Tools help before changing its rules.
Does blocking AI training also block AI search?
No, and this is the key point. Training and search are controlled by different tokens, and the vendors say so directly.
OpenAI's documentation says blocking GPTBot signals that content should not be used for training, while OAI-SearchBot controls appearance in ChatGPT search. It also says that ChatGPT-User does not decide search inclusion, so search opt-outs should target OAI-SearchBot.
Google is similar. Its documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Apple says pages that disallow Applebot-Extended can still appear in search results, because that token only governs how already-crawled data is used.
The reverse also holds. If you block a search bot such as OAI-SearchBot or Claude-SearchBot, you may stay out of those assistants' answers, which costs you visibility.
What about user-triggered fetchers?
These bots act for a person, not for the vendor's index. OpenAI says ChatGPT-User does not crawl automatically and that robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt because a user initiated the request. Meta says meta-externalfetcher may bypass it too.
Anthropic's article says disabling Claude-User may reduce visibility in user-directed web search. It gives no robots.txt example for it, so test any rule rather than assuming it works. If you truly need to stop these fetches, robots.txt alone may not be enough.
What should a robots.txt look like to appear in AI answers but opt out of training?
This version keeps search and answer crawlers welcome and blocks the training tokens documented above. Paste it into your existing file, and keep any sitemap line you already have.
# Training opt-out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# Search and answer crawlers stay open
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Applebot
Allow: /
User-agent: meta-webindexer
Allow: /
Applebot is worth a note. Apple says its data may also help train Apple models, and that Applebot-Extended is the opt-out. Disallowing Applebot-Extended is therefore the right lever, not blocking Applebot.
What if you want to allow everything?
If you are happy for every crawler to read your public pages, you need very little. An empty Disallow means nothing is blocked.
User-agent: *
Disallow:
Sitemap: https://www.example.co.uk/sitemap.xml
Replace the sitemap address with your own. Keep private areas such as checkout or account pages out of reach with Disallow rules, and remember that robots.txt is a request, not a security control.
How do you check your changes worked?
Three habits save most of the pain.
- Test the file. A single typo in a User-agent line can silently ignore a whole group. Our free Robots & Sitemap Checker reads your live file and flags problems.
- Allow time. OpenAI and Perplexity both say changes can take about 24 hours, and Meta says crawlers may cache robots.txt for up to 24 hours.
- Check your firewall. A robots.txt that allows a bot is useless if your security layer blocks it. Anthropic notes that IP blocking may not reliably opt a site out, and Perplexity suggests matching both user-agent and published IP ranges.
For the wider picture on crawling, rendering and indexing, see our technical SEO guide. If a term here is unfamiliar, the glossary explains it in plain English.
In short
- Training and search are separate: opt out of training tokens like GPTBot, ClaudeBot and Google-Extended without touching OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot.
- User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt, so do not rely on it to stop them.
- Test your file with the Robots & Sitemap Checker and recheck vendor documentation, because tokens and policies change.