Robots.txt for AI Crawlers: What to Allow, What to Block

Your robots.txt for AI crawlers has become a business decision, not just a technical file. The same file that once only talked to Googlebot now decides whether ChatGPT, Claude, Perplexity and Gemini can read, quote and link your pages. The format itself has not changed: it is still the Robots Exclusion Protocol standardised as RFC 9309, with user agents, Allow and Disallow lines. What changed is the number of agents, and the fact that one vendor can run several bots with different purposes.

Which AI crawlers exist and what do they do?

Most AI vendors now separate training, search indexing and user-triggered fetching into different user agents. The table summarises the main ones as documented by each vendor.

Sources: OpenAI crawler documentation and Google's list of common crawlers. Names change occasionally, so check each vendor's page before editing the file. Our glossary entries on AI crawlers and OpenAI crawlers keep the definitions short.

Should you block AI crawlers?

For most businesses that want customers, no: block training if you must, keep search crawlers open. The trade-off depends on what you sell:

• Brands, SaaS, e-commerce, local businesses: AI answers are an acquisition channel. Blocking search crawlers removes you from it. Allowing training crawlers also helps future models know your brand.
• Publishers living from page views or subscriptions: blocking training crawlers protects content value; keeping search crawlers open still brings citations and links.
• Sites with private or licensed content: block broadly and enforce at the server level, since robots.txt is a request, not an access control.

What does a recommended robots.txt look like?

Explicit groups per user agent, with search bots allowed and a clear choice for training bots. Example for a business that wants AI visibility but prefers not to feed training data:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml

Remove the second group if you are happy for your content to be used in training. Our free robots.txt generator builds the file, and the robots.txt checker shows which agents a live file actually allows.

Which mistakes block AI visibility by accident?

The most common one is a blanket rule copied from a forum. Frequent errors:

• "User-agent: * / Disallow: /" left from staging, which blocks every bot including Googlebot.
• Blocking OAI-SearchBot while meaning to block GPTBot, which removes you from ChatGPT search instead of training only.
• A CDN or firewall bot-protection rule that returns 403 to AI user agents regardless of robots.txt. Check your server logs.
• Blocking the CSS and JavaScript needed to render the page, so crawlers see an empty shell.
• Important content only loaded client-side; many AI crawlers do not execute JavaScript, so ship the main text in the HTML.

Is robots.txt enough to get cited?

No, it only opens the door. Once crawlers can read your pages, citation depends on whether those pages answer questions clearly, are structured for extraction and are confirmed by other sources. Clean structured data for AI search, answer-first sections and an optional llms.txt file help the next step.

Conclusion

Treat AI crawlers as separate decisions: open the search and user-fetch bots, choose deliberately for training bots, and verify that your server applies the same policy. Then measure whether the open door turns into citations. Sorank tracks your brand in ChatGPT, Claude and Gemini every week and publishes content structured to be cited: start with a 3-day free trial.