Quick answer: Allow the search crawlers that put your pages into AI answers, such as OAI-SearchBot, Googlebot, Claude-SearchBot and PerplexityBot, and decide separately on training crawlers such as GPTBot, ClaudeBot and Google-Extended. User-triggered fetchers such as ChatGPT-User may ignore robots.txt. Blocking a search crawler can remove your pages from that tool’s answers.
Key takeaways
- AI crawlers fall into 3 types: search crawlers, training crawlers and user-triggered fetchers, and each type needs its own robots.txt decision.
- OpenAI lists 4 user agents as of November 19, 2026: OAI-SearchBot for ChatGPT search, GPTBot for training, ChatGPT-User for user actions and OAI-AdsBot for ad landing pages.
- Allow the search crawlers of the AI tools where you want customers to find you. Blocking GPTBot is not the same as blocking ChatGPT search.
- Google says Googlebot rules control Search, including AI Overviews and AI Mode. Google-Extended is a separate token that does not affect Google Search inclusion.
- OpenAI says robots.txt changes can take about 24 hours to apply to its search results.
- Zyte found 12.7% of 11,100 popular sites name at least one AI crawler in a Disallow rule, and a robots.txt file is a request, not a lock.
What are AI crawlers?
AI crawlers are automated programs that fetch web pages for AI products such as ChatGPT, Claude, Perplexity and Gemini, and each one announces itself with a user agent name. Your robots.txt file tells the well-behaved ones which parts of your site they may fetch.
Google Search Central’s crawler overview, accessed on November 19, 2026, splits crawlers into common crawlers, special-case crawlers and user-triggered fetchers. It says common crawlers always respect robots.txt rules for automatic crawls, while a user-triggered fetcher is started by a person using a tool or product feature. AI companies use the same split in practice, which is why a single “block AI” switch does not exist.
What are the 3 types of AI crawlers?
The 3 types of AI crawlers are search crawlers, training crawlers and user-triggered fetchers, and the type decides what happens to your business when you allow or block a crawler in robots.txt.
The 3 types of AI crawlers and what each one does are listed below.
- Search crawlers: bots that find and index pages so an AI tool can show and cite them in answers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Training crawlers: bots that collect content that may be used to train AI models, such as GPTBot, ClaudeBot and CCBot. Google-Extended is a robots.txt token that controls Gemini training and grounding.
- User-triggered fetchers: bots that open a page only when a person asks a question or starts an action, such as ChatGPT-User, Claude-User and Perplexity-User.

The table below lists the AI crawlers named in each company’s official documentation as of November 19, 2026.
| Crawler | Company | Type | robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Honors rules |
| GPTBot | OpenAI | Training | Honors Disallow |
| ChatGPT-User | OpenAI | User-triggered | May not apply |
| Googlebot | Search | Honors rules | |
| Google-Extended | Training and grounding | Token only | |
| Claude-SearchBot | Anthropic | Search | Honors rules |
| ClaudeBot | Anthropic | Training | Honors rules |
| Claude-User | Anthropic | User-triggered | Honors rules |
| PerplexityBot | Perplexity | Search | Allow advised |
| Perplexity-User | Perplexity | User-triggered | Generally ignores |
| CCBot | Common Crawl | Open dataset | Honors Disallow |
The table follows OpenAI, Google Search Central, Anthropic, Perplexity and Common Crawl documentation, and OpenAI also lists OAI-AdsBot, which only visits pages submitted as ChatGPT ads.
Which AI crawlers should your business allow in robots.txt?
Your business should allow the search crawlers of the AI tools where customers might find you, and decide on training crawlers based on how much you want to keep your content out of model training. User-triggered fetchers need no special rule, because robots.txt does not reliably control them.
Google adds one condition that applies to every AI crawler decision: a page must be indexed and eligible to be shown in Google Search with a snippet to be a supporting link in AI Overviews or AI Mode, and indexing and serving are not guaranteed. Allowing a crawler opens the door and does not promise a citation. Crawler access is the first layer of SEO services that aim to get a business recommended in AI answers.
Should you allow AI search crawlers?
Yes, allow AI search crawlers if you want your pages to appear in that tool’s answers, because blocking them can remove you. OpenAI says OAI-SearchBot surfaces websites in ChatGPT search features and that opted-out sites will not appear in ChatGPT search answers, though they can still show as navigational links.
Anthropic says that blocking Claude-SearchBot means your content is not indexed for search, which may reduce its visibility. Perplexity’s documentation recommends allowing PerplexityBot in robots.txt so your site can show up in Perplexity results, and says PerplexityBot is not used to gather content for AI foundation models.
Should you block training crawlers like GPTBot?
No, blocking GPTBot is not the same as blocking ChatGPT search, because OpenAI uses OAI-SearchBot for ChatGPT search features and GPTBot for content that may be used in training its generative AI foundation models. Blocking GPTBot with a Disallow rule signals that your content should not be used for training.
Anthropic describes ClaudeBot the same way: blocking it means a site’s future materials should be excluded from its AI model training datasets. Common Crawl’s CCBot builds an open repository of web crawl data, and Common Crawl gives the opt-out as a User-agent: CCBot group with Disallow: /. Whether to block a training crawler is a business choice about your content, and our guide to the shared AI training opt-out covers how to decide.
Should you allow user-triggered fetchers like ChatGPT-User?
Yes, leave user-triggered fetchers such as ChatGPT-User alone if you want ChatGPT users to be able to read your pages on request, because OpenAI says robots.txt rules may not apply to ChatGPT-User. A Disallow line is therefore not a dependable block for that bot.
Behavior differs by company. Anthropic says Claude-User honors robots.txt, and Perplexity says Perplexity-User generally ignores robots.txt rules because a person requested the fetch. OpenAI also says ChatGPT-User is not used to decide whether content appears in ChatGPT search.
What about Bingbot and Microsoft Copilot?
Microsoft’s controls for Copilot answers are page-level meta tags rather than a robots.txt token, according to its Bing Webmaster Blog post of September 22, 2023. The post says NOARCHIVE keeps a page out of Bing Chat answers, and NOCACHE limits what Bing may show to the URL, title and snippet.
That post does not mention robots.txt, and the Microsoft pages we reviewed on November 19, 2026 do not name a separate AI training crawler token for Bing. Bing Webmaster Tools added an AI Performance report in public preview on February 10, 2026, which shows how often your content is cited as a source in AI-generated answers, so use it to see the effect of any change.
How do you write robots.txt rules for AI crawlers?
You write robots.txt rules for AI crawlers by adding one User-agent group per crawler name, followed by Allow or Disallow lines, in the robots.txt file at the root of each domain and subdomain you own. Anthropic notes that a block applies to the host that serves the file, so repeat it for each subdomain.
The 4 steps to set your AI crawler rules are listed below.
- Copy your current file: open yourdomain.com/robots.txt and save a backup before you change it.
- Add a group for each search crawler: write the user agent name, then Allow: / if you want that tool to crawl your whole site.
- Add a group for each training crawler you want to block: write the user agent name, then Disallow: /.
- Save and test the file: reload yourdomain.com/robots.txt and read it back, then check your wildcard rules as described later in this guide.
The example below is an illustration, not a recommendation for every business. It allows 2 search crawlers and blocks 2 training crawlers, using the Disallow syntax that Anthropic and Common Crawl document.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Anthropic also supports the non-standard Crawl-delay line, for example Crawl-delay: 1, to slow its crawlers down instead of blocking them. Google-Extended works as a token in the same file, and Google says it has no separate user agent string.
Do robots.txt rules control Google AI Overviews and AI Mode?
Googlebot rules in robots.txt are the crawl control for Google Search, which includes the AI Overviews and AI Mode features, according to Google Search Central’s AI features page. Google-Extended is separate and does not control AI Overviews or AI Mode.
Google’s common crawlers page says Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It adds that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal. Our comparison of SEO vs AEO vs GEO explains how these Google features fit into a wider plan.
To limit what Google shows from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex controls, not to Google-Extended. Google also says it intends to support the AIPREF training preference signal in early 2027. For the selection side, read how AI Overviews choose which businesses to cite.
How long do AI crawler changes in robots.txt take to apply?
OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust for search results. The other providers covered here did not state a timing in the official pages we read on November 19, 2026, so plan for a delay and do not judge a change after a single day.
Google says recrawling a page after you change its preview controls can take anywhere from days to months. A new robots.txt rule is therefore a slow lever, and the right way to measure it is over weeks, using your search and referral data.
What share of websites block AI crawlers?
12.7% of the 11,100 popular landing pages in Zyte’s State of Web Access 2026 report name at least one AI crawler in a Disallow rule, and GPTBot is the most named at 8.4%. Zyte published the report on September 7, 2026.

The table below shows the 4 AI crawlers Zyte reported by name that are covered in this guide.
| Crawler | Type | Sites naming it |
|---|---|---|
| GPTBot | Training | 8.4% |
| CCBot | Open dataset | 7.3% |
| ClaudeBot | Training | 6.3% |
| Google-Extended | Training and grounding | 5.6% |
Zyte studied the world’s most popular landing pages, so these shares describe large sites, not typical small business sites.
The same Zyte report found that 27.2% of sites block all automated traffic with a wildcard rule and 1.8% explicitly allow AI, which puts the combined effective block rate at 39.9%. A wildcard rule such as User-agent: * with Disallow: / blocks AI crawlers too, which is a common way for a business to block AI search crawlers by accident.
How do you check which AI crawlers can reach your site?
You check which AI crawlers can reach your site by reading your robots.txt file, then testing the pages themselves, because a firewall or bot-protection rule can stop a crawler before it ever reads robots.txt. Anthropic warns against IP-based blocking for this reason, as it can stop its bots from reading your robots.txt at all.
The 5 checks to run on your own site are listed below.
- Read your robots.txt file: open yourdomain.com/robots.txt and look for any User-agent group that names an AI crawler.
- Find the wildcard group: check whether User-agent: * carries a Disallow: / rule that blocks AI crawlers you meant to allow.
- Inspect your key pages in Google Search Console: use the URL Inspection tool to see the HTML Googlebot receives, as Google’s AI features page advises.
- Search your server logs: look for each user agent name, such as OAI-SearchBot or ClaudeBot, and note the response codes.
- Ask your host or developer about bot rules: confirm that no firewall or content delivery network rule blocks the AI search crawlers you allowed.

Is robots.txt enough to control AI crawlers?
No, robots.txt is not enough to control AI crawlers, because it is a request that only crawlers choosing to follow it will honor. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Search Engine Journal reported on August 14, 2026 that TollBit’s State of the Bots report for the first half of 2026 found ChatGPT-User reached disallowed pages on more sites than any other tracked AI bot. That matches OpenAI’s own note about user-initiated visits. Firewall and bot-management settings are a separate layer that can contradict your file, and they decide what a crawler can actually fetch. Fixing mistakes like these is part of the website mistakes that stop service businesses getting leads.
Frequently asked questions
Does blocking Google-Extended hurt my Google rankings?
No, blocking Google-Extended does not hurt your Google rankings. Google’s common crawlers page says Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal in Google Search. It controls Gemini training and grounding only.
Does allowing an AI crawler mean my business will be cited?
No, allowing an AI crawler does not mean your business will be cited. Google says indexing and serving are not guaranteed, and a page must be indexed and eligible to show with a snippet to be a supporting link in AI Overviews or AI Mode. Crawler access is a requirement, not a result.
Is a Disallow line enough to stop ChatGPT-User?
No, a Disallow line is not enough to stop ChatGPT-User. OpenAI’s crawler overview says robots.txt rules may not apply to ChatGPT-User because the actions are user-initiated. If a page must stay private, put it behind a login instead of relying on robots.txt.
Does ClaudeBot respect robots.txt?
Yes, ClaudeBot respects robots.txt. Anthropic says ClaudeBot, Claude-User and Claude-SearchBot all honor robots.txt do-not-crawl directives, and that a User-agent: ClaudeBot group with Disallow: / at the top level of your site blocks ClaudeBot. It also supports the Crawl-delay extension.
How often should you review your AI crawler rules?
Review your AI crawler rules every quarter, because the official lists change. OpenAI documents 4 user agents on November 19, 2026, and Google says it intends to support the AIPREF training preference signal in early 2027. Re-check each company’s crawler page before you edit your file.
What should your business do about AI crawlers in robots.txt?
Your business should allow the AI search crawlers of the tools where customers look for you, make a deliberate choice on training crawlers, and treat robots.txt as one layer of control, not a lock. Test the result on your own pages every quarter.
If you want this checked alongside your rankings, our local SEO services and SEO services for business owners cover crawl access, indexing and AI search visibility together, and you can see how SERP Master Agency works with businesses in the US, UK, Canada, Australia and New Zealand. For a tool-by-tool view of recommendations, read how to get your business recommended by ChatGPT.
Want your AI crawlers set up correctly?
SERP Master Agency reviews your robots.txt file, your firewall settings and your key pages, then tells you which AI crawlers can reach your business and what to change. Tell us about your business and the areas you serve, and we will show you where the gaps are.
Sources
- OpenAI, “Overview of OpenAI Crawlers”, accessed November 19, 2026. https://developers.openai.com/api/docs/bots
- Google Search Central, “Overview of Google crawlers and fetchers”, accessed November 19, 2026. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Google Search Central, “Google’s common crawlers”, accessed November 19, 2026. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Google Search Central, “AI features and your website”, accessed November 19, 2026. https://developers.google.com/search/docs/appearance/ai-features
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, accessed November 19, 2026. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity, “Perplexity crawlers: PerplexityBot and Perplexity-User”, accessed November 19, 2026. https://docs.perplexity.ai/guides/bots
- Common Crawl, “CCBot”, accessed November 19, 2026. https://commoncrawl.org/ccbot
- Bing Webmaster Blog (Microsoft), “Announcing new options for webmasters to control usage of their content in Bing Chat”, September 22, 2023. https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat
- Bing Webmaster Blog (Microsoft), “Introducing AI Performance in Bing Webmaster Tools Public Preview”, February 10, 2026. https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
- Zyte, “State of Web Access 2026”, September 7, 2026. https://www.zyte.com/blog/robots-txt-ai-crawlers/
- Search Engine Journal, “OpenAI Says robots.txt May Not Apply To ChatGPT’s Fetch Bot”, August 14, 2026. https://www.searchenginejournal.com/openai-says-robots-txt-may-not-apply-to-chatgpts-fetch-bot/585864/

Suraj Rana is the founder of SERP Master Agency and Dental Master Media, based in Chandigarh, India. He has worked in SEO for over 9 years, helping local businesses, clinics, ecommerce stores and marketing agencies win more calls, leads and sales from Google and AI search. His team delivers SEO services, Google Business Profile optimisation, paid ads and white-label SEO for agencies. Every SERP Master guide is written to give business owners a plain-English answer: what to fix first, what it costs and what results to expect.


