Preferences
Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website. More information
Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website. More information
Crawlability is how easily bots can discover and fetch the pages of a site. What blocks crawlers, how to test it and how to fix it for Google and the AIs.
Tools
Summary
Crawlability is the ability of search engine and AI bots to discover the pages of a website and fetch their content without obstacles. A page that cannot be crawled cannot be indexed, ranked or cited.
Want to go further? Ask
Crawlability describes how easily a crawler, such as Googlebot, Bingbot or an AI bot like GPTBot, can find the pages of a site by following links and sitemaps, then download their content. It depends on the site's structure, its server, its rules for bots and the way its pages are built. It is the first condition of organic visibility: before any question of content or authority, the page has to be reachable.
Crawlability is often confused with crawling itself. Crawling is the process the search engine runs; crawlability is the property of your site that makes that process easy or difficult. It is also distinct from indexability: a page can be crawlable but excluded from the index by a noindex directive or a canonical pointing elsewhere.
Links and structure. Crawlers move from link to link. Pages that receive no internal links, called orphan pages, are hard to discover. Pages buried six or seven clicks from the home page are crawled less often. A clear hierarchy and solid internal linking are the foundation.
Rules for bots. The robots.txt file can block entire sections, sometimes by mistake. Login walls, aggressive firewalls or bot protection can also stop crawlers that are welcome.
Server health. Slow responses, timeouts and 5xx errors lead Google to reduce its crawl rate to protect your server. Long redirect chains and broken links waste crawl requests.
Rendering. When content and links only appear after JavaScript runs, crawlers have to render the page. Google does it, with a delay, but many AI crawlers fetch the raw HTML only and miss content generated in the browser.
Start with Google Search Console: the page indexing report lists URLs that could not be fetched, and the crawl stats report shows response times, status codes and the share of requests by bot type. Use the URL Inspection tool to see the rendered HTML Google gets for a given page.
Then crawl the site yourself with a desktop crawler, or with Sorank's free website crawlability AI audit, to find orphan pages, broken links, redirect chains and pages blocked by robots.txt. Compare the URLs found by the crawl with those in your XML sitemap: gaps on either side reveal structural problems.
Link every important page from at least one relevant page, and keep key pages within three clicks of the home page. Publish an up-to-date XML sitemap containing only canonical, indexable URLs. Review robots.txt so that it blocks only what must stay out, such as internal search or infinite filter combinations. Fix broken links and replace redirect chains with direct links.
On the technical side, serve important content and links in the initial HTML, through server-side rendering or static generation, rather than relying only on client-side JavaScript. Keep server response times low and monitor 5xx errors. For large sites, manage the crawl budget by reducing low-value URLs.
The classic error is a robots.txt rule left over from a development environment that blocks the whole site, or a directory that holds CSS and JavaScript files needed for rendering. Others include navigation built only with JavaScript click events instead of real links with an href attribute, endless URL parameters generated by filters and sorting, and links pointing to redirected or deleted pages after a migration.
For SEO, crawlability is a prerequisite: pages that are not crawled are not indexed and cannot rank, and new or updated content takes longer to appear. For GEO, it is just as decisive. AI assistants rely on their own crawlers and on search indexes. GPTBot, ClaudeBot or PerplexityBot have to be allowed in robots.txt, and they need content in plain HTML to read it. Our page on AI crawlers details which bots to allow.
Crawlability is the ease with which bots can discover and fetch your pages. It rests on clear internal links, a reasoned robots.txt, a healthy server and content available without heavy JavaScript. It comes before indexing, ranking and AI citations, so it deserves a regular check.
To go further, read our pages on crawling and crawl budget. Reference sources: Google's overview of its crawlers and Google's JavaScript SEO basics.
Sources
Crawlability is whether bots can reach and download a page. Indexability is whether the page, once crawled, is allowed and judged worthy of being stored in the index. A page blocked in robots.txt is not crawlable; a crawlable page with a noindex tag is not indexable.
Most AI crawlers fetch the raw HTML and do not execute JavaScript the way Google's renderer does. Content and links that only appear after client-side rendering may be invisible to them. Serving important content in the initial HTML is the safest approach.
Check the crawl stats and page indexing reports in Google Search Console. A rise in server errors, long response times, many "Discovered, currently not indexed" URLs or important pages missing from the index are typical signs of crawlability problems.
THEY TRIED SORANK BEFORE YOU
“I tested Sorank on a real client site and it's insane. It does everything an agency would bill you $1500 a month for. It's going into production on that client now.”
Guillaume Guersanguersanguillaume.com“We are an Ads agency and we wanted to get our SEO back on track once and for all. Sorank gave us a clear view of our site, helped us prioritize optimizations, and properly structure our content. The tool is easy to use, well designed for a team, and very practical for centralizing tracking without having to juggle multiple platforms. A very solid foundation to start fresh with our SEO.”
Gaspard MadouiAds Agency
“I just let the Sorank software do its thing, and now my DR is 25 and my page is ranking higher.”
Lucas Flandrefoundertrace.fr“I specialized in GEO thanks to Sorank. The GEO Leaderboard allows me to show my clients their position vs competitors on ChatGPT. It’s a real commercial differentiator.”
MélanieSEO Freelance“Great service, works well for our website and helps us understand how to improve it to rank better on Google and elsewhere. Congrats, keep going, I recommend it.”
Hugo T.E-commerce merchant
