Technical GEO in 2026: AI crawler access, server rendering, llms.txt, structured data, speed and freshness, with a checklist to get cited by AI.

On this page
In short
Technical GEO is the set of site settings that let AI assistants such as ChatGPT, Claude, Perplexity and Gemini fetch, read and quote your pages. It comes down to six checks: let the AI search crawlers in (robots.txt, firewall, CDN), serve your main content in the initial HTML because most AI crawlers do not execute JavaScript, keep pages fast and error-free, add structured data that matches the visible text, publish an llms.txt file as a low-cost extra, and signal freshness with accurate dates and sitemaps. None of this guarantees a citation, but any failure on these points removes your page from the candidates.
Want to go further? Ask
Quick quiz
How should you get mentioned by ChatGPT and other AI assistants?
Sorank matches your answers
Test it for 3 days on your own site and see where you stand on Google, ChatGPT and the other AIs.
Start 3-day free trial3 days free, then stop whenever you want. 90-day results-or-refund guarantee.
Technical GEO is the infrastructure side of generative engine optimization: everything that happens before an AI model judges whether your content is worth citing. Content and authority decide who gets quoted, but a page that an AI crawler cannot reach, or reaches as an empty JavaScript shell, never enters the race. Google itself sums up the baseline for its AI features: a page must be indexed and eligible for a snippet, and there are no additional technical requirements for AI Overviews or AI Mode. The other AI assistants run their own crawlers, with their own limits, and that is where most sites lose visibility without knowing it. This guide covers each layer, from bot access to freshness, with a checklist you can run in an afternoon.
Technical SEO makes a site crawlable and indexable for Googlebot. Technical GEO extends the same work to a dozen other bots that behave differently: they render less, crawl less often and follow fewer rules. The overlap is large (status codes, canonicals, sitemaps, speed), so a site with clean technical SEO already has most of the foundation. The differences sit in four places.
| Topic | Technical SEO (Googlebot) | Technical GEO (AI crawlers) |
|---|---|---|
| Bots to manage | Googlebot, Bingbot | OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and others |
| JavaScript | Rendered, with a delay | Not executed by most AI crawlers |
| Unit of value | The page and its ranking | The passage that gets quoted in an answer |
| Opt-out logic | Block or noindex a page | Separate choices for training, search and user-triggered visits |
| Measurement | Search Console | Server logs and prompt tracking |
The practical consequence: a site can rank well on Google and still be invisible to ChatGPT or Claude, because the content is built in JavaScript or because a firewall rule blocks unknown bots.
Each AI company now runs several bots, and each one has a different job. Blocking the wrong one can remove you from AI answers while you only meant to opt out of model training.
According to OpenAI's crawler documentation (checked October 2026), OAI-SearchBot decides whether your pages appear in ChatGPT search answers, GPTBot collects content that may be used to train its models, and ChatGPT-User fetches a page when a user asks for it. Disallowing GPTBot keeps your content out of training without removing you from ChatGPT search. Changes to robots.txt take about 24 hours to apply to search results, and robots.txt rules may not apply to ChatGPT-User because the visit is triggered by a person.
Anthropic follows the same three-way split, as its help center explains: ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for pages fetched during a conversation. Blocking Claude-SearchBot or Claude-User reduces your visibility in Claude's answers. Anthropic also warns that blocking its IP addresses is not a reliable opt-out, since the bot may then fail to read your robots.txt at all.
Google's AI Overviews and AI Mode use the regular Googlebot index. The Google-Extended token only controls the use of your content for Gemini training and grounding in some other Google systems, and Google states that it does not affect inclusion or ranking in Google Search. Perplexity uses PerplexityBot for its index. For a ready-made configuration per bot, see our guide to robots.txt rules for AI crawlers and the list of AI crawlers and their user agents.
| Goal | Allow | Optional to block |
|---|---|---|
| Be cited in ChatGPT | OAI-SearchBot, ChatGPT-User | GPTBot (training only) |
| Be cited in Claude | Claude-SearchBot, Claude-User | ClaudeBot (training only) |
| Be cited in Perplexity | PerplexityBot | None |
| Appear in AI Overviews and AI Mode | Googlebot | Google-Extended (no effect on Search) |
Watch out
Robots.txt is only half the check. Since July 2025, Cloudflare blocks AI crawlers by default on many configurations, and other firewalls and bot-protection tools do the same. Look at your CDN and security settings, then confirm in your server logs that the AI search bots receive a 200 status code.
Mostly no, and this is the most common technical GEO failure. A Vercel analysis of its network (December 2024) found that none of the major AI crawlers rendered JavaScript: the bots of OpenAI, Anthropic, Meta, ByteDance and Perplexity download the raw HTML and stop there. ChatGPT and Claude did fetch JavaScript files but did not execute them. Gemini is the exception because it relies on Googlebot's rendering infrastructure. The same study measured the scale of this traffic: in one month, GPTBot made 569 million requests and Claude 370 million, against 4.5 billion for Googlebot.
If your product descriptions, prices, articles or FAQ are injected by a React, Vue or Angular app after page load, AI crawlers see an empty template. The fix is to serve the essential content in the initial HTML response:
A quick test: open the page source (not the inspector) or fetch the URL with curl. If your main paragraph is missing from the raw HTML, AI crawlers cannot quote it. Our guide to JavaScript SEO covers the rendering options framework by framework, and the free AI crawlability audit checks what a bot receives.
AI assistants that browse in real time fetch several pages per question and work under tight time limits. A slow or broken page is skipped and another source is quoted. The Vercel study also found that ChatGPT and Claude crawlers hit 404 pages in about 34% of their fetches, against 8% for Googlebot, often by requesting outdated URLs. Three points matter most:
Tables deserve a special mention: comparison data in an HTML table is easy to extract, while the same data in an image or a canvas chart is lost for every AI crawler.
Structured data helps machines identify what a page is about (an article, a product, an organization, a person) and connect it to known entities. Google is clear that no special schema is needed for its AI features, and that markup must match the visible text. Its value for technical GEO is clarity, not a ranking boost. The types worth implementing on most sites:
Keep JSON-LD in the initial HTML, not injected by a tag manager, for the same JavaScript reason as above. The structured data audit for LLMs checks your markup against what a model can read, and our article on structured data for AI search details which types still matter.
Tip
Write the same key facts (company name, founding year, prices, locations) identically in your visible text, your schema and your official profiles. Consistent facts across sources make it easier for a model to describe your brand without mixing it up with another one.
The llms.txt proposal, published by Jeremy Howard in September 2024, is a Markdown file at the root of a site that lists its most useful pages with a short description, so that an AI agent finds the right content without parsing navigation menus. It is a proposal and not a formal standard, it controls neither access nor indexing, and Google does not list it among the requirements for its AI features. Treat it as a low-cost extra: it takes less than an hour, it helps agents and coding assistants that look for it, and it forces you to decide which pages represent your site. You can draft one with the llms.txt generator, and offer clean Markdown versions of key pages if your stack allows it.
AI assistants that search the web favor recent and dated sources for any question where time matters (prices, tools, rules, statistics). Content freshness is partly an editorial matter, but the signals are technical:
| Check | How to verify | Priority |
|---|---|---|
| AI search bots allowed in robots.txt | Read robots.txt for OAI-SearchBot, Claude-SearchBot, PerplexityBot | High |
| No firewall or CDN block | Server logs show 200 responses for AI bots | High |
| Main content in raw HTML | View source or curl the URL | High |
| Key pages indexed by Google and Bing | Search Console and Bing Webmaster Tools | High |
| 200 status, single-hop redirects | Crawl with an SEO crawler | Medium |
| Semantic headings, lists and tables | Read the HTML outline | Medium |
| Organization and Article schema | Schema validator | Medium |
| Visible dates and accurate lastmod | Compare page, schema and sitemap | Medium |
| llms.txt at the root | Open /llms.txt | Low |
Run it once on your 20 most important pages, then after every redesign, CMS change or security update. AI crawler logs are the only direct proof that the bots reach your pages; everything else is inference.
Technical fixes open the door; the result shows up in AI answers. Two measurements close the loop. First, server logs: visits from OAI-SearchBot, Claude-SearchBot and PerplexityBot should rise on the pages you fixed. Second, citations: track a fixed set of questions your customers ask and check, week after week, whether your brand is named and which sources are quoted instead. Patterns from large samples help to set expectations, as in our ChatGPT citation study.
Sorank, our software, handles that second part with AI mention tracking: 10 to 50 tracked prompts depending on the plan, on ChatGPT, Claude and Gemini, plus Grok from the Pro plan, with your visibility rate and the competitors cited in your place.

Once the technical base is clean, the next limit is content: pages that answer questions clearly, with sources. The article generator writes and publishes optimized articles for Google and AI on autopilot, with internal links and sources.
Technical GEO decides whether AI assistants can read your site; content and authority decide whether they quote it. Fix access, rendering and structure first, then measure citations. Plans and the 3-day free trial are on the pricing page.
Blocking GPTBot only tells OpenAI not to use your content for training its models. It does not remove your site from ChatGPT search answers, which depend on OAI-SearchBot. If you want to be cited in ChatGPT, keep OAI-SearchBot allowed whatever you decide for GPTBot.
No. Google states that AI Overviews and AI Mode have no technical requirements beyond being indexed and eligible for a snippet in Search. llms.txt can help other AI agents find your key pages, but it has no effect on Google rankings or AI Overviews.
AEO (answer engine optimization) and GEO describe the same goal: being quoted in answers generated by AI assistants. Technical GEO, sometimes called technical AEO, covers the infrastructure part of that work: bot access, rendering, speed, structured data and freshness. The content and authority parts come on top of it.
THEY TRIED SORANK BEFORE YOU
