Crawl
A crawl is a pass in which a bot fetches web pages and follows their links, whether a search engine discovering content or an SEO tool auditing a site.
What is a Crawl?
A crawl is a single run in which an automated bot visits web pages, downloads their content and follows the links it finds to discover more pages. Search engines perform crawls continuously to find new and updated content for their index. SEO professionals also run their own site crawls with tools that imitate search engine bots to audit a website.
The word is used both ways in SEO conversations. “Google hasn’t crawled the new page yet” refers to search engine crawling. “Run a crawl before the migration” refers to an audit crawl you control. Both answer the same question: what can a bot reach and read on this site?
How it works
Search engine crawl
- The crawler starts from known URLs, sitemaps and links from other sites.
- It checks robots.txt to see what it is allowed to fetch.
- It requests each URL, records the response and, for Google, may render JavaScript.
- It extracts links and adds new URLs to its queue.
- Fetched content is passed on for indexing.
SEO audit crawl
- Configure the tool: start URL, user agent, JavaScript rendering, crawl limits.
- The tool follows internal links (and optionally sitemaps) across the site.
- It reports issues by URL.
What an audit crawl typically finds
| Issue | Why it matters |
|---|---|
| 4xx and 5xx errors | Broken experiences, wasted crawling |
| Redirect chains | Slower crawling, diluted signals |
| Missing or duplicate titles and descriptions | Weak search snippets |
| Canonical and noindex conflicts | Pages dropping from the index |
| Orphan or deep pages | Hard for bots and users to find |
Examples
Illustrative scenarios:
- Before migrating its blog to a new CMS, a SaaS company crawls the old site to build a full URL list for its redirect map, then crawls the new site to confirm every old URL redirects correctly.
- A monthly crawl of a security vendor’s site reveals 120 internal links pointing to a retired product page that now redirects twice. Updating the links removes the chain.
- A crawl with JavaScript rendering disabled shows that a DevTools docs site’s navigation links are invisible without JavaScript, explaining why some pages were rarely crawled.
Best practices
- Crawl as both a desktop and smartphone user agent, since Google indexes mobile-first.
- Compare rendered and raw HTML crawls on JavaScript-heavy sites.
- Connect Search Console and analytics data to crawl tools to prioritize issues.
- Throttle crawl speed on fragile servers.
- Save crawls so you can compare before and after changes.
Common mistakes
- Crawling only from the homepage and missing orphan pages; also crawl your XML sitemaps.
- Fixing every warning instead of prioritizing by impact.
- Forgetting to exclude staging or logged-in areas, or crawling production too aggressively.
Crawls and AI search
AI engines rely on crawls too: GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot fetch pages to train models or to answer questions with live sources. Many of these crawlers do not render JavaScript, so content that only appears after client-side rendering may be invisible to them. A crawl with JavaScript disabled is a quick proxy for what they see. See how AI engines cite sources.
Related terms
Sources
Frequently asked questions
What is a site crawl in SEO?
A site crawl is when you use a crawler tool, such as Screaming Frog, Sitebulb, Lumar or the crawlers in Ahrefs and Semrush, to fetch every reachable page on your site. The output lists URLs with status codes, titles, meta tags, canonicals, links, depth and more, so you can find broken links, redirects, duplicates and missing metadata.
How often should I crawl my website?
Run a full crawl at least monthly for active sites, before and after migrations or redesigns, and after large content releases. Large sites often schedule weekly crawls of key sections and monitor changes automatically. Save each crawl so you can compare results and spot regressions between releases.
Can I ask Google to crawl my site?
You can request indexing for individual URLs with the URL Inspection tool in Search Console and submit XML sitemaps to help discovery. Google decides when and how much to crawl. Requesting a recrawl does not guarantee indexing. If important pages are rarely crawled, improve internal links to them and check for server or robots.txt issues.