Crawl

SEO, AEO & GEO Glossary

A crawl is a pass in which a bot fetches web pages and follows their links, whether a search engine discovering content or an SEO tool auditing a site.

What is a Crawl?

A crawl is a single run in which an automated bot visits web pages, downloads their content and follows the links it finds to discover more pages. Search engines perform crawls continuously to find new and updated content for their index. SEO professionals also run their own site crawls with tools that imitate search engine bots to audit a website.

The word is used both ways in SEO conversations. “Google hasn’t crawled the new page yet” refers to search engine crawling. “Run a crawl before the migration” refers to an audit crawl you control. Both answer the same question: what can a bot reach and read on this site?

How it works

Search engine crawl

  1. The crawler starts from known URLs, sitemaps and links from other sites.
  2. It checks robots.txt to see what it is allowed to fetch.
  3. It requests each URL, records the response and, for Google, may render JavaScript.
  4. It extracts links and adds new URLs to its queue.
  5. Fetched content is passed on for indexing.

SEO audit crawl

  1. Configure the tool: start URL, user agent, JavaScript rendering, crawl limits.
  2. The tool follows internal links (and optionally sitemaps) across the site.
  3. It reports issues by URL.

What an audit crawl typically finds

IssueWhy it matters
4xx and 5xx errorsBroken experiences, wasted crawling
Redirect chainsSlower crawling, diluted signals
Missing or duplicate titles and descriptionsWeak search snippets
Canonical and noindex conflictsPages dropping from the index
Orphan or deep pagesHard for bots and users to find

Examples

Illustrative scenarios:

  • Before migrating its blog to a new CMS, a SaaS company crawls the old site to build a full URL list for its redirect map, then crawls the new site to confirm every old URL redirects correctly.
  • A monthly crawl of a security vendor’s site reveals 120 internal links pointing to a retired product page that now redirects twice. Updating the links removes the chain.
  • A crawl with JavaScript rendering disabled shows that a DevTools docs site’s navigation links are invisible without JavaScript, explaining why some pages were rarely crawled.

Best practices

  • Crawl as both a desktop and smartphone user agent, since Google indexes mobile-first.
  • Compare rendered and raw HTML crawls on JavaScript-heavy sites.
  • Connect Search Console and analytics data to crawl tools to prioritize issues.
  • Throttle crawl speed on fragile servers.
  • Save crawls so you can compare before and after changes.

Common mistakes

  • Crawling only from the homepage and missing orphan pages; also crawl your XML sitemaps.
  • Fixing every warning instead of prioritizing by impact.
  • Forgetting to exclude staging or logged-in areas, or crawling production too aggressively.

AI engines rely on crawls too: GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot fetch pages to train models or to answer questions with live sources. Many of these crawlers do not render JavaScript, so content that only appears after client-side rendering may be invisible to them. A crawl with JavaScript disabled is a quick proxy for what they see. See how AI engines cite sources.

Sources

Frequently asked questions

What is a site crawl in SEO?

A site crawl is when you use a crawler tool, such as Screaming Frog, Sitebulb, Lumar or the crawlers in Ahrefs and Semrush, to fetch every reachable page on your site. The output lists URLs with status codes, titles, meta tags, canonicals, links, depth and more, so you can find broken links, redirects, duplicates and missing metadata.

How often should I crawl my website?

Run a full crawl at least monthly for active sites, before and after migrations or redesigns, and after large content releases. Large sites often schedule weekly crawls of key sections and monitor changes automatically. Save each crawl so you can compare results and spot regressions between releases.

Can I ask Google to crawl my site?

You can request indexing for individual URLs with the URL Inspection tool in Search Console and submit XML sitemaps to help discovery. Google decides when and how much to crawl. Requesting a recrawl does not guarantee indexing. If important pages are rarely crawled, improve internal links to them and check for server or robots.txt issues.

Technical SEOCrawlingSite Audit

Do not let AI keep
recommending someone else

Start your 7-day free trial and get your AI Visibility Score in about a minute. See exactly where you stand, where competitors are beating you, and the ranked fixes to get into the answer. Cancel anytime before the trial ends.

7-day free trial. Cancel anytime before it ends. Trusted by 500+ B2B SaaS teams.

Partnered with
Powered by