On this page
It’s good to allow search-focused AI crawlers such as OAI-SearchBot and PerplexityBot to crawl your site in robots.txt. Also, try to block crawlers you don’t want accessing or using your content, such as training-focused bots like GPTBot or CCBot.
This way, you can improve your website’s visibility in AI-generated answers with just a few lines of code in your robots.txt.
The right rules for robots.txt depend on whether you want to support AI search visibility. Also, you have to decide whether you want to allow your content to be used for model training.
Your robots.txt file lets you manage these permissions by setting simple rules for individual crawlers.
In this guide, we’ll explain the purpose of each AI model and whether you have to allow them to crawl your pages or block their access.
Key Takeaways
-
Allow AI Search Bots: Let bots like OAI-SearchBot and PerplexityBot crawl your content for AI search.
-
Block Training Bots: Block GPTBot if you don’t want OpenAI to collect your content for model training. Block CCBot if you don’t want your content included in Common Crawl’s web archive.
-
Set Specific Rules: Create separate robots.txt rules for each bot to control access.
What Is robots.txt?
robots.txt is a plain-text file in the root directory of your website. It tells compliant web crawlers which parts of the site they can or cannot access.
For example, here is the content of a robots.txt:
User-agent: GPTBot
Disallow: /
This instruction tells OpenAI’s GPTBot not to crawl any part of your website.
The User-agent directive identifies the crawler. The Disallow command specifies which paths GPTBot should avoid.
On the other hand, if you use the Allow directive, you can explicitly permit access to a path.
Using these instructions, you can let AI crawlers understand your access preferences as the site’s owner.
Important Note: Different companies have different rules. This means some bots that fetch pages at a user’s request may not follow robots.txt rules.
If you want AI models to mention your brand or choose your content as a direct answer to users’ questions, allow crawlers to discover, retrieve, and cite your website.
Remember that AI crawlers have different purposes, including:
- Building search indexes
- Collecting information for model development
- Retrieving individual webpages when someone asks a question
Keep in mind that robots.txt is not a security tool. It asks compliant crawlers to follow your instructions but does not prevent unauthorized bots from accessing public pages. Never use it to protect confidential information.
robots.txt vs. llms.txt
Many website owners think they only need to focus on llms.txt to ensure AI accessibility and visibility. However, there is a big difference between their purposes:
| File | Main purpose |
|---|---|
| robots.txt | Allows/blocks crawling and determines access preferences. |
| llms.txt | Provides a structured guide to important website content and resources. |
In fact, llms.txt is a file for presenting important website information in a way that AI systems can easily process.
However, AI platforms are not required to support or use it.
Key AI Bots for AI Visibility
Here are the most important AI bots that you need to consider when editing your robots.txt:
1. Search and citation crawlers
These crawlers are the most important bots for AI-powered search:
- OAI-SearchBot: Helps make website content available in ChatGPT Search.
- Claude-SearchBot: Helps Claude improve its search results.
- PerplexityBot: Builds Perplexity’s search index.
2. User-triggered retrieval crawlers
The following crawlers retrieve specific pages when users request information:
- ChatGPT-User
- Claude-User
- Perplexity-User
Therefore, you can’t assume that an Allow or Disallow rule will have the same effect across all three services.
3. Training-related crawlers
These crawlers have a different purpose:
- GPTBot
- ClaudeBot
- Google-Extended
- Applebot-Extended
An important distinction is that Google-Extended and Applebot-Extended are robots.txt control tokens. So, don’t consider them separate HTTP crawling agents.
Also, do not confuse ClaudeBot with Claude-SearchBot. Anthropic introduces ClaudeBot as its model-development crawler and provides separate agents for search and user-requested retrieval.
Which Bots Should You Allow?
You should specifically allow some important bots to make sure they can crawl your pages. Here is a list of some of these bots:
| Crawler | Owner | Why allow it? |
|---|---|---|
| OAI-SearchBot | OpenAI | Helps ChatGPT find and cite your content |
| Claude-SearchBot | Anthropic | Supports Claude search results |
| PerplexityBot | Perplexity | Helps Perplexity index and cite your content |
| Googlebot | Supports Google Search and AI Overviews | |
| Bingbot | Microsoft | Supports Bing Search and Copilot |
| Applebot | Apple | Supports Apple search features |
Does allowing AI crawlers improve website visibility?
Even if you allow relevant crawlers, you can’t be sure about AI visibility. Using the Allow rule will remove a potential access barrier, but it can’t guarantee visibility in AI searches.
Many other factors help your content appear in AI answers, including:
- Content quality: Provide accurate, useful, and well-written information.
- Relevance: Match the topics and questions users search for.
- Crawlability: Make your pages accessible to search engines and AI crawlers.
- Indexability: Ensure search engines can index your content.
- Clear structure: Use descriptive headings, lists, and concise answers.
- Authority: Build trust through expertise and reliable sources.
- Freshness: Keep information accurate and up to date.
- Mentions and citations: Get referenced by trusted websites and sources.
More importantly, even if you specifically allow some bots, they might still be blocked by other tools like security plugins and tools. This usually happens with Anthropic bots, especially when you use security plugins like Jetpack for your WordPress website.
So, if Claude-SearchBot cannot access your website despite being allowed in robots.txt, check your server logs, firewall, CDN, and security plugin settings.
You need to identify the actual blocker before changing your security configuration.
Which Bots Should You Block?
Here are some bots that you need to block based on your policies, like privacy:
| Crawler | Owner | Why block it? |
|---|---|---|
| GPTBot | OpenAI | Collects content for AI training |
| ClaudeBot | Anthropic | Collects content for AI training |
| Google-Extended | Controls use of content for Gemini training and grounding | |
| Applebot-Extended | Apple | Controls use of content for Apple AI training |
| CCBot | Common Crawl | Collects content for its open web archive |
| Bytespider | ByteDance | Associated with AI training data collection |
| meta-externalagent | Meta | Collects content to improve Meta AI products |
When should you block bots?
Some businesses should be more careful when allowing bots to crawl their website. For example:
If you publish proprietary research or commercially sensitive material, you can’t simply allow all the crawlers to access your content.
Moreover, if you do not want crawlers to use your publicly available content for model training, you should block them.
An important matter is your licensing agreements. If they restrict certain forms of content reuse, you have to comply with these restrictions when allowing access to bots in your robots.txt.
Sample robots.txt Files for Better Results in AI Answers
Here are some useful examples of robots.txt files for different purposes:
Sample 1: Allow search & citation but block training
If you want your content to become discoverable for AI search services but avoid being used for model-training collection, use a robots.txt like this:
# OpenAI: Allow search, block training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
# Anthropic: Allow search and user retrieval
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
# Perplexity: Allow search
User-agent: PerplexityBot
Allow: /
# Google: Restrict Gemini training and grounding
User-agent: Google-Extended
Disallow: /
# Apple: Restrict model training
User-agent: Applebot-Extended
Disallow: /
This configuration allows:
- OpenAI’s search crawler
- Anthropic’s search and user-requested retrieval crawlers
- Perplexity’s search crawler.
Also, it restricts GPTBot, ClaudeBot, Google’s documented Gemini training and grounding uses, and Apple’s documented foundation-model training use.
Remember that some bots, like OpenAI’s ChatGPT-User and Perplexity-User, might ignore robots.txt rules.
So, this configuration does not guarantee control over user-triggered retrieval.
Sample 2: Allow selected AI crawlers
As a website owner, you might want to allow some AI services while restricting others.
For example, you might want to allow ChatGPT and Perplexity discovery while blocking Anthropic’s crawlers and AI training collection.
Here is the robots.txt file you have to create for this scenario:
# OpenAI
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
# Anthropic
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
# Google
User-agent: Google-Extended
Disallow: /
# Apple
User-agent: Applebot-Extended
Disallow: /
### ```
Sample 3: Block all relevant AI crawlers
This is also a robots.txt file that blocks all the documented AI models from crawling your website:
```text
# OpenAI
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
# Anthropic
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
# Perplexity
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
# Google
User-agent: Google-Extended
Disallow: /
# Apple
User-agent: Applebot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Common Crawl
User-agent: CCBot
Disallow: /
## ```
Final Thoughts
robots.txt is a plain-text file that lets you manage the accessibility of your site for AI crawlers and bots. It’s located in the root directory of your site and can allow or block documented AI bots. You have to edit this file according to your policy and requirements. If you specifically allow AI search crawlers to access your site, you can increase the AI visibility of your content. Of course, this doesn’t guarantee AI mentions, but it ensures that AI crawlers are not blocked by robots.txt. On the other hand, some crawlers don’t follow robots.txt rules, so you can’t be sure that they are blocked.




