GEO

AI Crawlers in robots.txt: What to Allow or Block

AI crawlers can affect how your website appears in AI-powered search results and how companies use your content to train their models. By configuring your robots.txt file, you can decide which crawlers can access your website and which ones should be blocked. In this guide, you'll learn what the main AI bots do and how to create the right rules for your website.

AI crawler table in MonoRanks
On this page

It’s good to allow search-focused AI crawlers such as OAI-SearchBot and PerplexityBot to crawl your site in robots.txt. Also, try to block crawlers you don’t want accessing or using your content, such as training-focused bots like GPTBot or CCBot.

This way, you can improve your website’s visibility in AI-generated answers with just a few lines of code in your robots.txt.

The right rules for robots.txt depend on whether you want to support AI search visibility. Also, you have to decide whether you want to allow your content to be used for model training.

Your robots.txt file lets you manage these permissions by setting simple rules for individual crawlers.

In this guide, we’ll explain the purpose of each AI model and whether you have to allow them to crawl your pages or block their access.

Key Takeaways

  • Allow AI Search Bots: Let bots like OAI-SearchBot and PerplexityBot crawl your content for AI search.

  • Block Training Bots: Block GPTBot if you don’t want OpenAI to collect your content for model training. Block CCBot if you don’t want your content included in Common Crawl’s web archive.

  • Set Specific Rules: Create separate robots.txt rules for each bot to control access.

What Is robots.txt?

robots.txt is a plain-text file in the root directory of your website. It tells compliant web crawlers which parts of the site they can or cannot access.

For example, here is the content of a robots.txt:

User-agent: GPTBot

Disallow: /

This instruction tells OpenAI’s GPTBot not to crawl any part of your website.

The User-agent directive identifies the crawler. The Disallow command specifies which paths GPTBot should avoid.

On the other hand, if you use the Allow directive, you can explicitly permit access to a path.

Using these instructions, you can let AI crawlers understand your access preferences as the site’s owner.

Important Note: Different companies have different rules. This means some bots that fetch pages at a user’s request may not follow robots.txt rules.

If you want AI models to mention your brand or choose your content as a direct answer to users’ questions, allow crawlers to discover, retrieve, and cite your website.

Remember that AI crawlers have different purposes, including:

  • Building search indexes
  • Collecting information for model development
  • Retrieving individual webpages when someone asks a question

Keep in mind that robots.txt is not a security tool. It asks compliant crawlers to follow your instructions but does not prevent unauthorized bots from accessing public pages. Never use it to protect confidential information.

robots.txt vs. llms.txt

Many website owners think they only need to focus on llms.txt to ensure AI accessibility and visibility. However, there is a big difference between their purposes:

File Main purpose
robots.txt Allows/blocks crawling and determines access preferences.
llms.txt Provides a structured guide to important website content and resources.

In fact, llms.txt is a file for presenting important website information in a way that AI systems can easily process.

However, AI platforms are not required to support or use it.

Key AI Bots for AI Visibility

Here are the most important AI bots that you need to consider when editing your robots.txt:

1. Search and citation crawlers

These crawlers are the most important bots for AI-powered search:

  • OAI-SearchBot: Helps make website content available in ChatGPT Search.
  • Claude-SearchBot: Helps Claude improve its search results.
  • PerplexityBot: Builds Perplexity’s search index.

2. User-triggered retrieval crawlers

The following crawlers retrieve specific pages when users request information:

  • ChatGPT-User
  • Claude-User
  • Perplexity-User

Therefore, you can’t assume that an Allow or Disallow rule will have the same effect across all three services.

These crawlers have a different purpose:

  • GPTBot
  • ClaudeBot
  • Google-Extended
  • Applebot-Extended

An important distinction is that Google-Extended and Applebot-Extended are robots.txt control tokens. So, don’t consider them separate HTTP crawling agents.

Also, do not confuse ClaudeBot with Claude-SearchBot. Anthropic introduces ClaudeBot as its model-development crawler and provides separate agents for search and user-requested retrieval.

Which Bots Should You Allow?

You should specifically allow some important bots to make sure they can crawl your pages. Here is a list of some of these bots:

Crawler Owner Why allow it?
OAI-SearchBot OpenAI Helps ChatGPT find and cite your content
Claude-SearchBot Anthropic Supports Claude search results
PerplexityBot Perplexity Helps Perplexity index and cite your content
Googlebot Google Supports Google Search and AI Overviews
Bingbot Microsoft Supports Bing Search and Copilot
Applebot Apple Supports Apple search features

Does allowing AI crawlers improve website visibility?

Even if you allow relevant crawlers, you can’t be sure about AI visibility. Using the Allow rule will remove a potential access barrier, but it can’t guarantee visibility in AI searches.

Many other factors help your content appear in AI answers, including:

  1. Content quality: Provide accurate, useful, and well-written information.
  2. Relevance: Match the topics and questions users search for.
  3. Crawlability: Make your pages accessible to search engines and AI crawlers.
  4. Indexability: Ensure search engines can index your content.
  5. Clear structure: Use descriptive headings, lists, and concise answers.
  6. Authority: Build trust through expertise and reliable sources.
  7. Freshness: Keep information accurate and up to date.
  8. Mentions and citations: Get referenced by trusted websites and sources.

More importantly, even if you specifically allow some bots, they might still be blocked by other tools like security plugins and tools. This usually happens with Anthropic bots, especially when you use security plugins like Jetpack for your WordPress website.

So, if Claude-SearchBot cannot access your website despite being allowed in robots.txt, check your server logs, firewall, CDN, and security plugin settings.

You need to identify the actual blocker before changing your security configuration.

Which Bots Should You Block?

Here are some bots that you need to block based on your policies, like privacy:

Crawler Owner Why block it?
GPTBot OpenAI Collects content for AI training
ClaudeBot Anthropic Collects content for AI training
Google-Extended Google Controls use of content for Gemini training and grounding
Applebot-Extended Apple Controls use of content for Apple AI training
CCBot Common Crawl Collects content for its open web archive
Bytespider ByteDance Associated with AI training data collection
meta-externalagent Meta Collects content to improve Meta AI products

When should you block bots?

Some businesses should be more careful when allowing bots to crawl their website. For example:

If you publish proprietary research or commercially sensitive material, you can’t simply allow all the crawlers to access your content.

Moreover, if you do not want crawlers to use your publicly available content for model training, you should block them.

An important matter is your licensing agreements. If they restrict certain forms of content reuse, you have to comply with these restrictions when allowing access to bots in your robots.txt.

Sample robots.txt Files for Better Results in AI Answers

Here are some useful examples of robots.txt files for different purposes:

Sample 1: Allow search & citation but block training

If you want your content to become discoverable for AI search services but avoid being used for model-training collection, use a robots.txt like this:

# OpenAI: Allow search, block training

User-agent: OAI-SearchBot

Allow: /

User-agent: GPTBot

Disallow: /

# Anthropic: Allow search and user retrieval

User-agent: Claude-SearchBot

Allow: /

User-agent: Claude-User

Allow: /

User-agent: ClaudeBot

Disallow: /

# Perplexity: Allow search

User-agent: PerplexityBot

Allow: /

# Google: Restrict Gemini training and grounding

User-agent: Google-Extended

Disallow: /

# Apple: Restrict model training

User-agent: Applebot-Extended

Disallow: /

This configuration allows:

  • OpenAI’s search crawler
  • Anthropic’s search and user-requested retrieval crawlers
  • Perplexity’s search crawler.

Also, it restricts GPTBot, ClaudeBot, Google’s documented Gemini training and grounding uses, and Apple’s documented foundation-model training use.

Remember that some bots, like OpenAI’s ChatGPT-User and Perplexity-User, might ignore robots.txt rules.

So, this configuration does not guarantee control over user-triggered retrieval.

Sample 2: Allow selected AI crawlers

As a website owner, you might want to allow some AI services while restricting others.

For example, you might want to allow ChatGPT and Perplexity discovery while blocking Anthropic’s crawlers and AI training collection.

Here is the robots.txt file you have to create for this scenario:

# OpenAI

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: GPTBot

Disallow: /

# Perplexity

User-agent: PerplexityBot

Allow: /

# Anthropic

User-agent: ClaudeBot

Disallow: /

User-agent: Claude-SearchBot

Disallow: /

User-agent: Claude-User

Disallow: /

# Google

User-agent: Google-Extended

Disallow: /

# Apple

User-agent: Applebot-Extended

Disallow: /

### ```

Sample 3: Block all relevant AI crawlers

This is also a robots.txt file that blocks all the documented AI models from crawling your website:

```text
# OpenAI

User-agent: GPTBot

Disallow: /

User-agent: OAI-SearchBot

Disallow: /

User-agent: ChatGPT-User

Disallow: /

# Anthropic

User-agent: ClaudeBot

Disallow: /

User-agent: Claude-SearchBot

Disallow: /

User-agent: Claude-User

Disallow: /

# Perplexity

User-agent: PerplexityBot

Disallow: /

User-agent: Perplexity-User

Disallow: /

# Google

User-agent: Google-Extended

Disallow: /

# Apple

User-agent: Applebot

Disallow: /

User-agent: Applebot-Extended

Disallow: /

# Common Crawl

User-agent: CCBot

Disallow: /

## ```

Final Thoughts

robots.txt is a plain-text file that lets you manage the accessibility of your site for AI crawlers and bots. It’s located in the root directory of your site and can allow or block documented AI bots. You have to edit this file according to your policy and requirements. If you specifically allow AI search crawlers to access your site, you can increase the AI visibility of your content. Of course, this doesn’t guarantee AI mentions, but it ensures that AI crawlers are not blocked by robots.txt. On the other hand, some crawlers don’t follow robots.txt rules, so you can’t be sure that they are blocked.

Sources

Common questions

Does robots.txt stop AI crawlers?

Not always. It asks compliant crawlers to follow your access rules, but it cannot force every bot to obey them.

Is there a robots.txt for AI?

No. You use your website’s regular robots.txt file to allow or block AI crawlers by name.

Is robots.txt illegal?

No. Using robots.txt is legal. It is a standard way to tell crawlers which pages they can access.

What is an AI crawler bot?

An AI crawler is a bot that visits websites and collects content for AI search, answers, or model training.

About one email a month

What changed in search and AI answers, from what we see in audits. No card, no pitch, one click to leave.