PerplexityBot

What is PerplexityBot?

PerplexityBot is the web crawler Perplexity uses to discover and index pages so they can be surfaced and linked as sources in Perplexity’s answers. It identifies itself with the token PerplexityBot and follows robots.txt. Perplexity states it is not used to collect content for training AI foundation models, which makes it a search crawler rather than a training crawler.

Perplexity is an answer engine that cites its sources inline, often several per answer. For B2B marketers that makes it one of the more measurable AI surfaces: if your page is a source, the link is visible and clickable.

How it works

Perplexity documents two agents:

Agent	Purpose	robots.txt
PerplexityBot	Indexes and links sites in search results	Followed
Perplexity-User	Fetches pages when a user asks a question	Generally ignored

From Perplexity’s crawler documentation:

• The user agent string contains PerplexityBot/1.0 and a link to perplexity.ai/perplexitybot. Version numbers may change.
• IP ranges are published at perplexity.ai/perplexitybot.json, and for Perplexity-User at perplexity.ai/perplexity-user.json.
• Perplexity-User “generally ignores robots.txt rules” because a person initiated the fetch. It is also not used for crawling or for training.

Perplexity’s crawling practices have been disputed. In August 2025 Cloudflare published a report alleging that Perplexity used undeclared crawlers to get around no-crawl directives. Perplexity rejected the characterization and argued that user-driven agents differ from bots. Treat claims from both sides as contested.

From SEO to AEO and GEO

The SEO predecessor is a search engine crawler: Googlebot or Bingbot deciding whether you can be found. PerplexityBot fills that role for Perplexity. The change is in what follows crawling. Perplexity runs searches, retrieves passages, then writes an answer with numbered citations. Your page competes to be one of a few cited sources, not one of ten blue links.

What carries over: crawlability, fast server responses, content in the initial HTML, clear headings and accurate titles. What matters more: self-contained passages that answer a specific question, recent and dated content, and corroboration from other sites that Perplexity may also retrieve for the same prompt.

Because Perplexity shows sources so prominently, it is also a useful diagnostic engine. If competitors or review sites are cited for your core category prompts and you are not, the gap usually points to missing pages, unclear passages or weak third-party coverage rather than a crawler problem.

What to do about it

1. Allow PerplexityBot in robots.txt for public pages.
2. Check CDN and WAF rules. Bot-management products may challenge or block it by default. Look for 403 or challenge responses in logs.
3. Verify by IP, not user agent alone.
4. Make passages quotable. Put the direct answer under a question-shaped heading, then the detail.
5. Track citations. Run a fixed set of buyer prompts in Perplexity and record which URLs it cites, including competitors and review sites.
6. Watch referral traffic from perplexity.ai in analytics; it is one of the clearer AI referral signals.

Common mistakes

• Blocking PerplexityBot to stop training when Perplexity says it is not a training crawler. The result is lost citations.
• Expecting robots.txt to stop Perplexity-User. Perplexity says it generally does not.
• Ignoring third-party pages. Perplexity often cites reviews, forums and comparisons, not only vendor sites.

Related terms

• AI Crawler
• OAI-SearchBot
• ClaudeBot
• AI Citation
• Crawl-to-Referral Ratio
• How AI engines cite sources

Sources

• Perplexity crawlers, Perplexity documentation
• Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives, Cloudflare blog
• Agents or bots? Making sense of AI on the open web, Perplexity blog