Crawl Stats
Crawl stats show how a search engine crawls your site: requests, download size, response times, status codes and crawler types, as in Search Console's report.
What are Crawl Stats?
Crawl stats are the data describing how a search engine crawler interacts with your website: how many requests it makes, how much data it downloads, how fast your server responds and what status codes it receives. Google Search Console’s Crawl Stats report shows this for Googlebot over roughly the last 90 days.
Crawl stats are the main window into whether Google can crawl your site efficiently. They help diagnose server issues, confirm that a migration is being processed, and detect crawl waste on large sites. Bing Webmaster Tools offers similar crawl information for Bingbot.
How to read the report
Path: Search Console > Settings > Crawl stats > Open report.
Main charts
- Total crawl requests: number of requests Googlebot made.
- Total download size: bytes downloaded.
- Average response time: time to fetch a resource; rising times can reduce crawling.
Host status
Shows whether Google had problems with robots.txt fetching, DNS resolution or server connectivity. Recent failures appear as warnings.
Breakdowns
| Breakdown | What to look for |
|---|---|
| By response | Spikes in 404, 5xx or 3xx responses |
| By file type | Unexpectedly heavy JavaScript, image or “other” requests |
| By purpose | Balance of discovery vs refresh |
| By Googlebot type | Smartphone vs desktop, image, AdsBot and others |
Examples
Illustrative scenarios:
- After a site migration, crawl requests jump and 301 responses dominate for several weeks, which is expected as Google processes redirects. If 404s spike instead, redirect mapping is broken.
- A SaaS docs site sees average response time rise sharply and crawl requests fall after moving to a new hosting plan. Fixing server performance restores crawling.
- A large marketplace sees most requests going to parameterized filter URLs. Log analysis confirms the waste, and robots.txt rules reduce it.
Best practices
- Check monthly and after any release that changes URLs, hosting or rendering.
- Watch host status first; robots.txt or DNS failures can stop crawling.
- Drill into examples by clicking a breakdown row to see sample URLs.
- Combine with server logs for full URL-level and non-Google crawler data.
- Annotate changes so you can connect trends to releases.
Common mistakes
- Panicking at normal fluctuations in crawl volume.
- Assuming more crawling always means better SEO.
- Ignoring robots.txt fetch errors, which can make Google pause crawling.
- Using the report on small sites as if crawl budget were a constraint.
Crawl stats and AI search
The Search Console report only covers Google’s crawlers. To see whether AI crawlers such as GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot visit your site, and which pages they fetch, use server or CDN logs. That data shows whether your robots.txt and firewall settings allow the AI engines you want to be cited by. See how AI engines cite sources.
Related terms
Sources
Frequently asked questions
Where do I find crawl stats in Google Search Console?
Open Search Console, go to Settings, and open the Crawl stats report. It is available for root-level properties (domain properties or URL-prefix properties at the root). The report covers roughly the last 90 days and shows trends plus breakdowns by response, file type, purpose and Googlebot type.
What does a drop in crawl requests mean?
It can mean Google sees less need to crawl (fewer changes, less demand), or that your server is slowing or returning errors so Google reduced its crawl rate. Check the host status section, average response time and the share of 5xx responses. A drop after removing many low-value URLs can be normal and healthy.
What is the difference between discovery and refresh crawls?
In the Crawl Stats report, discovery means Googlebot requested a URL it had not crawled before, and refresh means it recrawled a known URL. A site publishing a lot of new content should see discovery requests; if almost all crawling is refresh, new pages may not be found efficiently.