Search Console Crawl Stats: How Googlebot Visits Your Site

This guide uses Search Console Help: Crawl Stats report, and Google's crawling documentation: Optimize your crawl budget, Googlebot and Reduce the Google crawl rate. Quotes are from those pages.

Do you need this report?

Probably not often. Google calls it a report "aimed at advanced users" and says: "If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." Most business sites are well under that.

It is still worth a look in a few situations: after a site move or redesign, when the server has been slow or down, when new pages take a long time to show up in Google, or when your host tells you Googlebot is using a lot of resources. To find out whether a page is indexed, and why not, use the Page indexing report instead. Crawl Stats is about how Google's requests went, not about what ended up in the index.

Where to find it

In Search Console, open Settings (Property settings) and then Crawl stats. It is not in the left menu with the other reports.

The report "is available only for root-level properties": a Domain property such as example.com, or a URL-prefix property at the root, such as https://example.com. A property for a folder like https://example.com/blog/ does not get it. If you are not sure which kind you have, our Search Console guide explains the two property types.

The landing page shows charts for three totals and four breakdown tables:

• Total crawl requests, successful or not. Requests for images, CSS and scripts on your own site count; requests to other domains do not. Duplicate requests for the same URL are counted one by one.
• Total download size in bytes. A resource that many pages use, such as a stylesheet Google has cached, is only requested the first time.
• Average response time for everything fetched, with each linked resource counted separately.
• Breakdowns by response, file type, purpose and Googlebot type. Click any row for a chart and a list of example URLs, then click a URL for that one request: crawl time, response code and size.

On a Domain property with subdomains, a Hosts list lets you look at one subdomain at a time. Google shows "Only the top 20 child domains that received traffic in the past 90 days".

What gets counted

A few rules explain most of the surprises in this report:

• Real URLs, not canonicals. "All URLs shown and counted are the actual URLs requested by Google; data is not assigned to canonical URLs as is done in some other reports."
• Each redirect hop is a request. If page1 redirects to page2 and page2 to page3, that is three requests. Redirect responses are filed under "Other file type". Client-side (JavaScript) redirects are not counted.
• Only this domain. An image you load from another domain or a CDN hostname is not in your report. A parent domain does see its subdomains, but sibling subdomains do not see each other.
• Both protocols. The report for http://example.com also includes requests to https://example.com, though the example URLs only show the property's own protocol.
• Not every request. Google notes as a known issue that "some requests might not be counted", so small differences from your server logs are normal.
• Crawls that never happened. When robots.txt could not be fetched, Google counts the fetches it would have made. This is the usual reason the report shows far more requests than your logs.

Example URLs are "not comprehensive, but just a representative example". A URL missing from the list does not mean Google skipped it.

Host status: robots.txt, DNS and server

Host status says whether Google ran into availability problems in the last 90 days. There are three states: no significant problems in 90 days; at least one problem, but more than a week ago; or a problem within the last week. The last one is the one to act on. Ideally, Google says, "your host status should be Green."

Click it to see three charts, each with a dotted red line. Going over the line counts as an issue; Google's example is DNS resolution failing "for more than 5% of requests on a given day".

Category	What it tracks	Where to look
robots.txt fetching	Failed requests for your robots.txt file	Your server or firewall rules for /robots.txt
DNS resolution	Your DNS server did not recognise the hostname or did not answer	Your DNS provider or registrar
Server connectivity	The server did not respond, or sent an incomplete response	Your host: capacity, timeouts, firewall blocks

Why robots.txt matters so much here

You do not need a robots.txt file, but the request for it has to succeed. A 200 counts as success even if the file is empty or has errors. So does a 403, 404 or 410, which Google reads as "the file does not exist". A 429 or 5XX does not. Google keeps a successful copy for up to 24 hours. When the request fails, Google's documented sequence is:

• For the first 12 hours, Google stops crawling the site but keeps asking for robots.txt.
• From 12 hours to 30 days, it uses the last robots.txt it fetched successfully.
• After 30 days, if the homepage is available, Google acts as if there is no robots.txt; if it is not, Google stops crawling.

So a firewall or security plugin that blocks /robots.txt for Googlebot, or a server that returns a 500 for it, can stop crawling of the whole site. What to put in the file is covered in our robots.txt and XML sitemaps guide.

Crawl responses

This table splits requests by response type, as a share of requests, not URLs. If one URL got a 500 once and a 200 the next time, that is 50% each. Google's guidance: "Most responses should be 200 or other "Good"-type responses, unless you are doing a site reorganization or site move."

Google's group	Responses	What to do
Good	200, 301/308, 302/307, meta refresh ("Moved (other)"), 304 Not modified	Nothing. Change a 302 to a 301 if the move is permanent.
Possibly good	404 Not found	Often correct for pages that are gone. Fix the ones caused by your own broken links.
Bad	robots.txt not available, 401/407, 5XX, other 4XX, DNS errors, fetch errors, page could not be reached, timeouts, redirect errors	Fix where you can, starting with 5XX and robots.txt.

A few notes from Google on the bad group. Server errors "cause availability warnings and should be fixed if possible". "Page could not be reached" means the request never reached the server, so it will not be in your logs either. For 401 pages, either block them in robots.txt or decide whether they should be public; Google warns that letting Googlebot past a login "effectively removes the security of the page", because Googlebot can be faked. For 404s and redirect chains, see our guide to 404 pages and redirects.

File type, purpose and Googlebot type

File type shows the share of responses (not bytes) for HTML, images, JavaScript, CSS, PDF, JSON, feeds and so on. Redirects land in "Other file type", and failed requests in "Unknown (Failed)". When the site is slow, Google suggests checking here whether many small images or slow resources on another host are part of the load. You can click a type to compare its response time with its request count by date.

Crawl purpose has two values: Discovery, a URL Google had never crawled before, and Refresh, a recrawl of a known page. After you publish a batch of new pages or submit a sitemap, Google says "you should ideally see a bump in discovery crawls". For pages that change often but are not recrawled often enough, Google's advice is to make sure they are in a sitemap.

Googlebot type names the crawler:

• Smartphone and Desktop. Google says that for most sites Search "primarily indexes the mobile version", so "the majority of Googlebot crawl requests will be made using the mobile crawler".
• Image and Video, for Google's image and video crawlers.
• Page resource load: the images, CSS and scripts Google fetches to render a page before indexing it. An image fetched this way counts here, not under Image.
• AdsBot, which checks ad landing pages. Google says it crawls URLs "about every 2 weeks", and a spike usually means new Dynamic Search Ads targets.
• StoreBot, the product shopping crawler, and Other agent type for the rest.

When crawling spikes

Google lists three usual causes: you unblocked a large section of the site, added a large new section, or added many Dynamic Search Ads targets. Its crawl rate page adds site features that create lots of URLs: faceted navigation, sorting and filtering, and "A calendar with a lot of URLs for specific dates".

If the extra crawling is actually hurting the server, Google's steps are:

1. Find which crawler it is, in your server logs or the Googlebot type table.
2. For quick relief, block that crawler in robots.txt (this "can take up to a day to take effect"), or return 503 or 429 when the server is near its limit. A Retry-After header can tell Google when to come back.
3. Remove the block or stop the errors after two or three days, once Google's crawl rate has adjusted.

Do not keep it up longer. Google warns against returning 503 or 429 "for more than two or three days", and its crawl rate page says that if Googlebot sees these codes on the same URL for several days, "the URL may be dropped from Google's index." Lowering the crawl rate also means new pages are found later and changes show up later. For AdsBot, the fix is fewer ad targets, smaller batches or more server capacity, or the spike comes back two weeks later.

When crawling drops

Google says your crawl rate "should be relatively stable over the time span of a week or two". Its reasons for a sudden drop:

• A new or very broad robots.txt rule, including one that blocks CSS or JavaScript Google needs to understand the pages.
• Slower responses: "Googlebot will throttle back its requests to avoid overloading your server." Compare the Average response time chart with the dates of the drop.
• More server errors, which have the same effect.
• Content that changes rarely or "isn't very high quality", which Google may crawl less often.

Slow responses are often a hosting problem; our guide to hosting, caching and CDNs covers where that time goes. Note that "You can't tell Google to increase your crawl rate." For small and medium sites that are not fully crawled, Google's advice is to update your sitemaps and make sure you are not blocking pages.

Crawl budget, briefly

Crawl budget is "the set of URLs that Google can and wants to crawl". It depends on two things. The crawl capacity limit goes up when the site responds quickly and steadily, and down when it slows or returns 5xx or 429 responses. Crawl demand depends on things like the number of URLs Google knows about, how popular they are, and how often they change. Google counts each hostname as its own site, so www.example.com and shop.example.com have separate budgets.

Google's crawl budget guide is written for large sites: roughly "1 million+ unique pages" changing weekly, "10,000+ unique pages" changing daily, or sites where many URLs sit in "Discovered - currently not indexed". If your new pages are crawled the same day you publish them, Google says you do not need it: keeping the sitemap current and checking the Page indexing report is enough. Its one piece of advice that does apply to small sites is to avoid duplicate URLs, since time spent on them is time not spent on your real pages.

Frequently asked questions

Where is the crawl stats report in Search Console?

Under Settings (Property settings), then Crawl stats. It only appears for Domain properties and root-level URL-prefix properties.

Does a small website need to check crawl stats?

Rarely. Google says sites with fewer than a thousand pages should not need this report. Check it after a site move, an outage, or when new pages take a long time to be crawled.

Why are the crawl requests higher than my server logs?

Usually because your robots.txt was unavailable. Google counts the fetches it would have made, even though they never reached your server.

What does host status mean?

It shows whether Google had trouble with robots.txt fetching, DNS resolution or server connectivity in the last 90 days, and whether the latest problem was within the past week.

Can I make Google crawl my site more often?

No. Google says you can't ask for a higher crawl rate. Keep your sitemap up to date, do not block pages you want crawled, and keep the server fast and free of errors.

How do I slow Googlebot down?

In an emergency, return 500, 503 or 429 for a short time, or block the crawler in robots.txt, and undo it within two or three days. Doing it for longer can lead to pages being dropped from the index.

Keeping a site fast, reachable and easy for Google to crawl is part of our website development work. See all our services, or browse more guides.