Blocked by robots.txt: What the Search Console Status Means and How to Fix It

The blocked by robots.txt status appears in the Page indexing report of Google Search Console, among the reasons why pages are not indexed. Google's own Page indexing report documentation describes it in one line: the page was blocked by your site's robots.txt file. Behind that line are two very different situations, a healthy block and an accidental one, plus a confusing cousin, "Indexed, though blocked by robots.txt". This guide shows how to tell them apart and fix each one without breaking anything else. If you are new to the file itself, start with our definition of robots.txt.

What does "Blocked by robots.txt" mean in Search Console?

Your robots.txt file sits at the root of the host (example.com/robots.txt) and tells crawlers which paths they may fetch. When Googlebot finds a URL, through a link or a sitemap, it checks that file first. If a Disallow rule matches the path for its user agent, it does not fetch the page. Search Console then lists the URL under "Blocked by robots.txt".

Two consequences follow. Google cannot see the content, the title, the internal links or any meta tag on the page. And because it never reads the page, it cannot see a noindex tag either. The status is listed as a reason for non-indexing, so for most sites it is informational: Google itself notes that URLs blocked by robots.txt are probably blocked on purpose.

The problem starts when the list contains pages you want on Google: product pages, articles, category pages, or a whole site after a migration. In that case the block is a direct cause of a website not showing up on Google, and it is one of the fastest technical issues to fix.

"Blocked by robots.txt" vs "Indexed, though blocked by robots.txt"

The second status looks contradictory, and it is the one that worries site owners most. Google's introduction to robots.txt explains it: robots.txt is not a mechanism for keeping a page out of Google. If other pages link to a disallowed URL, Google can still index the URL itself, without crawling it. The result then shows the address with no description, because Google never read the text.

The key is to decide first what you want for each URL: indexed or not indexed. Robots.txt answers a different question (crawled or not crawled), and mixing the two is the source of most errors.

Why are your pages blocked by robots.txt?

Most accidental blocks come from a handful of causes. Check them in this order:

• A staging rule pushed to production. Disallow: / protects a test site, then ships with the launch and blocks everything.
• A CMS setting. WordPress has a "Discourage search engines from indexing this site" option; other platforms have equivalent toggles that edit robots.txt or add headers.
• A rule broader than intended. Disallow: /p blocks /products/ and /pricing/ as well as /p/. Robots.txt rules match the start of the path.
• Wildcards. * matches any sequence of characters and $ marks the end of a URL. Disallow: /*? blocks every URL with a parameter, which can include paginated or filtered pages you want indexed.
• Blocked resources. Blocking /assets/ or /wp-includes/ hides the CSS and JavaScript Google needs to render the page, which degrades how it understands your content.
• A migration. Paths change, the old robots.txt stays, or the new host serves its own default file.
• The wrong host. Each subdomain and protocol has its own file: rules on www.example.com do not apply to shop.example.com.

Remember that Google follows the most specific rule (the longest matching path), and when an Allow and a Disallow are equally specific, the least restrictive one wins. A targeted Allow: /blog/ can therefore reopen a folder under a broader block.

How to fix "Blocked by robots.txt" step by step

1. Export the affected URLs

In Google Search Console, open Indexing, then Pages, and click the "Blocked by robots.txt" row. Export the list and sort it by folder. You will usually see a pattern within a minute: one directory, one parameter, one subdomain.

2. Separate deliberate blocks from mistakes

Mark each pattern as "keep blocked" or "should be on Google". Carts, checkouts, account pages, internal search results and admin areas belong in the first group. Anything that answers a search query, earns links or appears in your XML sitemap belongs in the second. A URL that is both in the sitemap and blocked is a contradiction to resolve.

3. Find the rule that matches

Open the URL Inspection tool on one example URL: it states whether crawling is allowed. Then read the live file, or paste the URL into our free robots.txt checker to see which line blocks it and for which user agent.

4. Edit the file

Remove or narrow the rule, or add a more specific Allow. Keep the change minimal and test it against both a URL you want to unblock and a URL that must stay blocked. If the file is a mess, rebuild a clean one with our robots.txt generator and compare before publishing.

5. Make Google read the new file

Google generally caches robots.txt for up to 24 hours, according to its robots.txt specification. To speed things up, open Settings, then the robots.txt report in Search Console (available for domain-level properties), and use "Request a recrawl" for the file. Then request indexing for your most important unblocked URLs in URL Inspection.

6. Validate and monitor

Click "Validate fix" on the issue page. Validation can take several days to a few weeks, and new URLs are checked as Google recrawls them. Our guide to Google indexing covers what happens after the crawl, when Google decides whether a page is worth keeping.

How to fix "Indexed, though blocked by robots.txt"

The fix depends on what you want for the page:

1. You want the page on Google. Remove the block. Google will crawl the page, read its content and show a normal result with a title and description.
2. You do not want the page on Google. Remove the block too, then add <meta name="robots" content="noindex"> in the head, or an X-Robots-Tag: noindex HTTP header for PDFs and other files. Once Google has recrawled the page and dropped it, you may block it again if crawl volume matters, though most sites never need to. Our noindex definition explains the options.
3. The page should not exist publicly. Protect it with a password or remove it (404 or 410). Neither robots.txt nor noindex is a security measure.

For faceted navigation with thousands of filter URLs, a robots.txt block can remain the right tool to save crawl budget. The "Indexed, though blocked" warning on a few of those URLs is then acceptable, as long as canonical versions of your categories are crawlable and indexed.

How to check that the fix worked

Three signals confirm a successful fix. URL Inspection on the live URL shows "Crawl allowed: Yes". The count of blocked pages in the Page indexing report falls over the following weeks. And the unblocked pages start to collect impressions in the Performance report. A quick pass with our indexability checker also catches the other blockers that often travel with robots.txt mistakes: a stray noindex, a wrong canonical or a redirect.

Monitoring matters because these errors come back. A theme update, a plugin or a deployment script can rewrite robots.txt without anyone noticing. Watching pages whose clicks and impressions drop suddenly is the fastest early warning.

The Sorank GEO and SEO dashboard connects to Search Console and shows those rising and falling pages in one view, so a block introduced by a deployment shows up as a cluster of declining URLs rather than as a slow, unexplained loss.

Robots.txt, AI crawlers and GEO

Robots.txt now has a second audience: the crawlers of AI assistants. OpenAI, Anthropic, Perplexity and Google each publish user agents (GPTBot and OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) that respect robots.txt. A rule meant for one of them, or a blanket User-agent: * block, decides whether ChatGPT, Claude or Perplexity can read your pages when they search the web to build an answer.

The same reasoning applies as with Google: decide per bot what you want. Blocking training crawlers while allowing search crawlers is a common compromise, and our guide to robots.txt for AI crawlers lists the user agents and the rules for each. The glossary entry on AI crawlers explains which bot does what.

To see the effect of these choices, track the prompts that matter to your business. AI mention tracking in Sorank shows whether ChatGPT, Gemini or Claude cite your site, and which competitors they cite instead.

The rule to remember

Robots.txt decides what crawlers may read; noindex decides what search results may show. Keep blocks for areas with no search value, never block a page you want indexed or de-indexed, and check the Page indexing report after every migration or deployment. Most "blocked by robots.txt" problems are fixed in minutes once the matching rule is found, and Sorank's Search Console dashboard helps you catch the next one before it costs traffic.