Preferences

Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website. More information (Cookie Policy)

Crawl Stats: How to Read the Google Search Console Report and Fix What It Shows

Learn to read every chart of the Crawl Stats report in Google Search Console, what it misses compared with server logs, and how to fix what it shows.

Published on
19/11/2026
Updated on
19/11/2026
Reading time
16 min
Crawl Stats: How to Read the Google Search Console Report and Fix What It Shows

Tools

On this page

In short

The Crawl Stats report in Google Search Console shows how Googlebot requests your URLs: how many requests it sends, how many bytes it downloads, how fast your server answers, and which status codes it gets back. Read the host status first, then the response codes, then the purpose and file type breakdowns. The report is a summary, not a log, so use it to find the problem, then confirm it in your server logs.

Want to go further? Ask

Quick quiz

Who should fix the technical side of your site?

Question 1 of 425%
What is happening today?

What the Crawl Stats report is, and who should open it

Crawl Stats is a Google-side view of Googlebot's activity on your site. It tells you what Googlebot did, not why your pages rank or do not rank. Google describes the report as aimed at advanced users, and a site with fewer than a thousand pages may not need it at all. If you run a small brochure site, you can skip it until something changes.

Google's public Search Console product page
Google's public Search Console page, the starting point for crawling and indexing tools.

Search Console is one product with many reports. Its public page covers crawling and indexing in general: sitemaps, URL Inspection, index coverage. Crawl Stats sits under Settings rather than in the main menu, which is why many people never open it.

Only root-level properties get the report

The report is available only for root-level properties: a Domain property, or a URL-prefix property built on the root of the site. A property for a subfolder, such as example.com/blog/, does not get it. The data covers only the domain you select. Requests to sibling subdomains do not appear in another subdomain's report, but a parent domain view includes all its child domains. Both http and https count in the same view.

Note

If your site uses language subdomains such as en.example.com and de.example.com, open each one separately. Each has its own data, and a broken German subdomain does not show up in the English view.

Who should use it

Open the report when one of these is true: you publish many pages, you changed your URL structure or your hosting, Googlebot seems to visit less than before, or pages are discovered but never indexed. A site that adds a few articles a month has little to learn from it except errors, and those errors are better found in a normal crawl of the site.

Open the report and read each chart

Open your property, go to Settings, find Crawl stats, and click Open report. At the top you get three charts: total crawl requests, total download size, and average response time. Below them you get the host status and four breakdowns: crawl responses, file type, crawl purpose, and Googlebot type. Every breakdown is a percentage based on request counts.

Google Search Console help page describing the Crawl Stats report
Google's help page for the Crawl Stats report.

Here is what each part counts and how to read it.

Part of the reportWhat it countsHow to read it
Total crawl requestsAll requests for your URLs, successful or not, including failed DNS, connection and redirect-loop attempts. Attempts blocked by robots.txt are counted too.A sudden drop or jump: check your robots.txt changes and your URL structure for the same dates.
Total download sizeBytes downloaded during crawling. A cached resource counts only on its first request.Bytes rising while requests stay flat means your pages got heavier.
Average response timeThe average time to fetch each resource, with each linked resource counted separately.Read the trend, not one value. Watch it after each deployment.
Host statusRobots.txt fetching, DNS resolution and server connectivity. Green means no significant issue in the past 90 days, yellow means issues more than a week old, red means issues in the last week.Any red or yellow item is a server or network problem before it is an SEO problem.
Crawl responsesThe share of responses by type, based on request counts.Track the share of bad responses over time. The raw count matters less.
File typeThe share of responses by type: HTML, image, JavaScript, CSS, PDF, JSON, video. Based on response count, not bytes.Shows where Googlebot spends its requests. Heavy JavaScript and CSS shares mean your pages depend on many resources.
Crawl purposeDiscovery (a URL Google had never crawled) or refresh (a recrawl of a known URL).See the section on discovery and refresh below.
Googlebot typeSmartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, other.On a site indexed mobile-first, Smartphone should be the largest group. A sudden shift deserves a check.

Percentages count requests, not URLs

Every breakdown counts requests. A URL that Googlebot fetches 50 times counts 50 times. So "40% of responses are 301" tells you how many requests hit a redirect, not how many URLs redirect. To learn which URLs are involved, you need your logs. The example URLs that Google shows are a sample, and Google says they are not comprehensive.

What the report does not show

Google's help page says that the report currently reports most crawl requests, but some requests might not be counted. It also says that failed requests that never reach your server will not appear in your logs. That is why the report shows DNS and connection failures that your logs cannot show, and why you should read both sources. Google does not state a retention period for the charts, so save your own copy each month, with a screenshot or a short note of the numbers, if you want history.

Blue and white isometric illustration of a bot moving between connected nodes toward a website
Every Googlebot request travels from Google to your pages, and every response is recorded.

Read the response codes, step by step

The response breakdown is the part of the report that most often leads to a fix. Read it in this order, and stop at the first step that shows a real problem.

Triage the crawl responses in five steps

  1. Check the host statusRed or yellow on robots.txt, DNS or connectivity means Googlebot could not reach your site at all. Fix this before anything else.
  2. Look at the bad responses5XX, other 4XX, robots.txt unavailable, timeouts and redirect errors. Note their share and whether it moves from week to week.
  3. Check the redirect shareMany 301 and 302 responses mean Googlebot spends requests on hops instead of pages.
  4. Confirm the 404sA 404 is fine if the page was removed on purpose. It is a problem if your internal links or sitemap still point to it.
  5. Pull the URLs from the logsTurn each bad category into a list of URLs, then fix the source of the links, not the report.

Good, possibly good, and bad responses

Google sorts the responses into three groups. The grouping is a first filter, not a verdict on each URL.

  • Good: 200, 301, 302, 304 and meta refresh.
  • Possibly good: 404.
  • Bad: robots.txt unavailable, 401 and 407, 5XX, other 4XX, DNS errors, fetch errors, timeouts and redirect errors.

A 401 counts as bad in the report, but it is correct if the page sits behind a login that should stay closed. Decide URL by URL. The report tells you which group to look at, and you decide what each response should be.

Redirects: the cost you do not see

A 302 counts as a good response, so a redirect share can look healthy while it still wastes crawling. In one analysis of a large site, published in April 2021 on Search Engine Journal by Tomek Rudzki of Onely, 16% of requests went to redirected pages, and the author advised checking each redirect hop.

Here is how a chain hurts. Suppose /blog/seo-tips redirects to /blog/seo-tips-2025, which redirects to /blog/seo-tips-guide. Googlebot needs three requests to reach the content, and every internal link that still points to the first URL repeats the cost. Point the first URL straight to the final address, and update the internal links so they stop using the old paths. The free redirect checker follows any URL's chain hop by hop, so you can see the path before you fix it. If you are not sure whether a redirect should be permanent, read our guide on 301 vs 302 redirects: a 301 says the move is permanent, a 302 says it is temporary.

Tip

Sort your redirect URLs by how often Googlebot requests them in your logs. Fix the five most requested chains first. They usually account for most of the redirect share.

5XX and 429: the signal that lowers the crawl rate

Google's guide for large sites says that when a site responds with server errors (5XX) or rate-limiting signals such as HTTP 429, Google lowers its crawl rate limit. So a spike in 5XX costs you more than the broken URLs. It can slow the crawl of your healthy pages too. The crawl budget of a site depends partly on how well the server answers, and 5XX responses shrink that part.

When you see a 5XX spike, look at when it happens. Errors at the same hour each day point to a backup job, a scheduled import or a traffic peak. Errors right after a release point to the release. Errors spread across the whole day point to capacity. Each pattern has a different fix, and none of them is solved by editing a single page.

Discovery, refresh and file type: what the ratios say

The crawl purpose breakdown splits requests into discovery, when Google sees a URL for the first time, and refresh, when it recrawls a URL it already knows. The ratio between them tells you what Google is spending its time on.

  • A high discovery share on a site that adds few pages means Google keeps finding new URLs. Look for URL generators: filter and sort parameters, internal search result pages, calendar archives, session IDs. Each one creates URLs that look new.
  • A high refresh share on pages that never change means Google keeps revisiting them. Check whether those pages return 304 Not Modified. Google says that returning 304 for an unchanged page tells it to reuse the copy it has. If the same pages return 200 with full bytes every time, your server sends the whole page on each visit.

One large site, one crawl day

7460pages crawled per day on average
35%of those crawls were discovery crawls, for URLs Google had never fetched

Source: Tomek Rudzki (Onely), Search Engine Journal, April 2021. One site of about 1 million pages, with a backlog of "Discovered, currently not indexed" URLs.

The same analysis made a useful rough check. At that pace, it estimated that the backlog would take roughly a year to crawl. That number is one author's calculation from one site, and the screenshots date from 2021, so do not copy it. The method still works on your own data: divide the number of URLs waiting to be discovered by the number of discovery crawls per day in your report. If the answer is measured in months, you have a crawl problem to fix before you publish more.

The file type breakdown counts responses, not bytes, so it shows how many requests go to each resource type, not how heavy they are. Redirects fall under "Other file type", and failed requests appear as "Unknown (Failed)". A growing share of Unknown (Failed) is a failure signal, not a file type, and you should treat it with the bad responses.

Crawl Stats, server logs and URL Inspection: three views of Googlebot

There are three ways to see what Googlebot did on your site: the Crawl Stats report, your server logs, and the URL Inspection tool in Search Console. Each answers a different question, and each has blind spots that the other two cover. For a real problem, use all three.

The Crawl Stats report

Principle: a summary of Googlebot's requests for a whole property, computed by Google. When to use it: for trends, host status, and the first overview of a problem. Advantages: it is free, it covers failures that never reach your server, and it adds purpose and Googlebot type, which logs do not give you directly. Limits: it is a summary, some requests may not be counted, and the example URLs are a sample. Example: the 5XX share rises for three days on one host. The report tells you the days and the host, and your logs tell you the URLs and the hours.

Server logs

Principle: the raw record your web server writes for every request it receives. When to use it: to get exact URL lists, status codes by hour, and the evidence behind a report trend. Advantages: complete for every request that reached your server, with the exact URL and status code. Limits: they do not contain requests that failed before reaching the server, the user agent can be faked, and most servers rotate or delete logs after a set period. Here is a command that counts the status codes of requests identified as Googlebot in a combined-format log:

grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

The command matches the user agent string, then counts the status code, which is the ninth field in the combined log format. If your server writes another format, change the field number. The result is a table of counts by status code, with the most frequent code first.

Warning

Anyone can send a request that claims to be Googlebot. Before you act on a suspicious spike, confirm that the IP addresses belong to Google. Google documents a reverse DNS check for this, and it is worth doing before you block or redirect anything based on the user agent.

The URL Inspection tool

Principle: a check on one URL, showing when Google last fetched it, what it received, and whether it is indexed. When to use it: to confirm one fix, for example whether Google fetched a corrected page after you changed it. Advantages: fast, exact and easy to read for a single URL. Limits: one URL at a time, so it cannot show a pattern across the site. In the analysis mentioned above, the tool showed that a main category page had not been crawled for three months, and the author used it as a stand-in for the logs on that page.

Crawl Stats report against server logs

Crawl Stats report
  • Counts failed DNS and connection attempts
  • Shows host status and Googlebot type
  • Samples example URLs, not all of them
  • Some requests may not be counted
Server logs
  • Every request that reached your server
  • Exact URL and status code for each request
  • Miss failed DNS and connection attempts
  • User agent can be faked
MethodQuestion it answersScopeBlind spot
Crawl Stats reportIs Googlebot healthy, and how is it behaving overall?The whole property, summarizedSampled example URLs, and some requests may not be counted
Server logsWhich exact URLs got which response, and when?Every request that reached your serverFailed DNS and connection attempts, and user agents that can be faked
URL InspectionWhen did Google last fetch this page, and what did it receive?One URLSays nothing about the rest of the site

A monthly review, step by step

One month of data tells you little. The report becomes useful when you compare it with your own normal, so keep a short record and review it on the same day each month. This is the routine that works for most sites.

The monthly review in six steps

  1. Save the monthScreenshot or note the three charts, the host status and the response breakdown.
  2. Compare with your baselineA change in total requests may be normal after a big publishing month. A new share of 5XX is always worth a look.
  3. Check in orderHost status first, then the bad responses, then the redirect share.
  4. Pull the URLsFor each bad category, take the top URLs from your logs for the same period.
  5. Fix the sourceThe internal links, the redirect rules, the server capacity or the robots.txt rule. Not the chart.
  6. Log the changeWrite down the date of each fix, and read the next month with that date in mind.

Signals that deserve action this month

  • Host status is yellow or red on any of the three checks
  • The 5XX share rises for several days in a row, or spikes at the same hour
  • Redirect responses grow after a site change that was meant to remove them
  • Discovery share stays high on a site that publishes little
  • A single bad day with no change in logs, which is noise until it repeats

Before and after a migration

Suppose you move your blog from /old-blog/ to /blog/ with 301 redirects. Right after the move, expect the redirect share to rise, because Googlebot still asks for the old paths. Over the following weeks, the share should fall as Google learns the new URLs. Expect the discovery share to rise too, since the new URLs are new to Google. If the redirect share stays high after several weeks, look for chains, for old internal links, or for redirects you forgot to add. Our guide to fixing broken links covers the link side of the same cleanup.

Common mistakes, and how to avoid them

Most mistakes come from reading one chart as a verdict. Each of the five below has a clear sign that shows up in the report, and a fix that you can check in your logs.

Reading total crawl requests as a score

What it is: assuming that more crawl requests mean better SEO. How to spot it: requests rise sharply after a release that added filter or sort options, and the new URLs carry parameters. How to fix it: check the discovery share and the URL patterns in your logs. Fix the URL generator: stop linking to parameter URLs, or canonicalize and block the ones that have no search value. A higher count of junk requests is a cost, not a gain.

Trusting the average response time

What it is: looking at the average and missing the spikes it hides. How to spot it: the average looks flat, but your logs show slow responses at the same hour every night. How to fix it: group your logs by hour and by URL, then compare the slow hours with your backups, imports and deployments. Fix the slow job or the slow template, then check the average again. The average is a trend line, and a trend line cannot tell you where the slowness starts.

Ignoring host status when your pages load fine

What it is: assuming the site is healthy because it opens in your browser. How to spot it: the host status shows yellow or red for robots.txt, DNS or connectivity, while your pages work for you. Googlebot reaches your site through different network paths, and a firewall, a CDN rule or a DNS record can block it without affecting your browser. How to fix it: check that your robots.txt file loads with a 200 response, using the robots.txt checker, then review your firewall and CDN rules for the dates when the status turned red.

Blocking the crawl, then reading the drop as progress

What it is: a robots.txt change that cuts crawl requests, followed by a report that looks calmer. How to spot it: total requests fall sharply on the day of a robots.txt change. How to fix it: a drop can be correct if you blocked junk paths, but check that your important sections still get crawled. Look at the logs for the sections that matter. If a page you need is blocked, our guide to the blocked by robots.txt status explains how to find the rule and what to change.

Mixing up crawling with indexing

What it is: treating a well-crawled page as an indexed page. How to spot it: Googlebot requests a URL regularly, but the URL does not appear in Google's results. A high crawl count does not prove that Google keeps the page. How to fix it: move from the Crawl Stats report to the page indexing reports and check the status of the URL. Our guide to crawled, currently not indexed explains the causes, and the guide to Google indexing covers the checks in order.

MistakeHow to spot itHow to fix it
Total requests read as a scoreRequests rise after URL parameters appearStop generating the junk URLs, then recount
Average response time aloneSlow hours hidden by a flat averageGroup logs by hour, match slow hours to jobs
Host status ignoredYellow or red while pages open in a browserCheck robots.txt, DNS, firewall and CDN rules
Crawl drop read as progressRequests fall on a robots.txt change dayConfirm important sections are still crawled
Crawling confused with indexingRegular crawls, URL missing from resultsCheck the indexing status of the URL

Where Sorank fits, and what to do next

Sorank's GEO / SEO dashboard puts your Search Console clicks, impressions, CTR and position next to the visits that AI assistants send you, read from Google Analytics. It does not show Googlebot's requests, so keep the Crawl Stats report for that job. The two tools answer different questions: the dashboard tells you what Google sends you, and Crawl Stats tells you how Google visits your server.

Tip

Before you spend time in your logs on a single page, check whether Google can index it at all with the free Google indexability checker. It answers the first question in seconds.

What you seeStart withThen open
Host status red or yellowThe robots.txt checker, then your DNS and hosting statusServer logs for the same dates
5XX share risingYour logs grouped by hour and URLServer capacity, or the template that fails
Redirect share highThe redirect checker on the top URLsRedirect chains and old internal links
Many URLs discovered, few indexedThe indexability checker on samplesOur log file analysis guide

The Crawl Stats report is a good first screen. It tells you which kind of problem to suspect and on which dates. The fix almost always lives in a list of URLs, and that list is in your logs. Start with the report, then go to the URLs.

Frequently asked questions

Where is the Crawl Stats report in Google Search Console?

Open your property, go to Settings, select Crawl stats, then click Open report. The report exists only for Domain properties and root URL-prefix properties, so a subfolder property will not show it.

Is a high number of crawl requests a good sign?

Not on its own. The total counts every request, including failed ones and attempts blocked by robots.txt, so it shows how much Googlebot visits, not how healthy those visits are. Read it next to the response codes and the average response time.

Can I rely on Crawl Stats instead of server logs?

Not on its own. Google says some requests may not be counted and that its example URLs are not comprehensive. Use the report for trends and failure categories, and your logs for exact URLs.

Get your content plan for 30 days for free

We analyse your competitors, your website and your AI visibility to create a tailored content plan for free.

Our free SEO tools

See all our free tools

Get your content plan for 30 days for free

We analyse your competitors, your website and your AI visibility to create a tailored content plan for free.

Man with dark hair and beard wearing a light brown shirt speaks in front of a microphone on a podcast or recording setup.Portrait of a man with short dark hair wearing a white shirt and dark jacket, looking directly at the camera with a neutral expression.Man with short dark hair, beard, and clear glasses wearing a black t-shirt with a white circular logo, standing in front of a stone wall.Celio fabianoSmiling young woman with long brown hair wearing a red top and necklace, outdoors in a tree-filled background.Profile photo of Xavier Breull, Sorank customer
Google
Solid yellow five-pointed star icon on transparent background.Solid yellow five-pointed star icon on transparent background.Solid yellow five-pointed star icon on transparent background.Solid yellow five-pointed star icon on transparent background.Solid yellow five-pointed star icon on transparent background.
5/5
+5,000
users

Testimonials from ambitious founders

0 to 10000 visitors in 5 months

Celio Fabianomonrobotlavevitre.frGoogleE-commerce

0 to 5000 visits in 8 months

Thibault BMsorank.comGoogleSoftware

0 to 1104 clicks in 4 months

Dominiqueflight-sensations-idf.comGoogleLeisure and aviation

0 to 163 clicks in 6 months

Coworking Pariscoworking-paris.frGoogleOffice broker

x92 clicks in 7 months

Guide Zanzibarguidezanzibar.comGoogleTravel