Crawlers & Access
Does Perplexity-User follow robots.txt?
Updated January 12, 2026GEO Expert Team
Summary
Perplexity's documentation notes that since the fetch is user-requested, this fetcher 'generally ignores robots.txt rules.' The user explicitly asked to access the content, so Perplexity honors that request.
Detailed Answer
Perplexity's documentation notes that since the fetch is user requested, this fetcher "generally ignores robots.txt rules."
Reasoning:
- User explicitly requested the content
- Similar to a user clicking a link
- Respects user intent over site preferences
Implications:
- Can't fully block Perplexity-User via robots.txt
- Would need other access controls (login, paywall)
- Most publishers accept this behavior
For context:
This is similar to how browsers fetch content when a user clicks a link—the user's explicit request takes precedence.
perplexityrobots-txtuser-agent