Crawlers & Access

Question 1

Which OpenAI crawlers matter for GEO?

Accepted Answer

OpenAI documents OAI-SearchBot (used to surface sites in ChatGPT search features), GPTBot (training crawler), and ChatGPT-User (user-initiated fetch, not general crawling). OAI-SearchBot is most important for visibility.

Question 2

If I block GPTBot, can I still appear in ChatGPT search?

Accepted Answer

Yes—OpenAI states the settings are independent; you can allow OAI-SearchBot while disallowing GPTBot. This lets you appear in search results without contributing to training data.

Question 3

What are Anthropic's crawlers?

Accepted Answer

Anthropic documents ClaudeBot (potential training data collection), Claude-User (user-initiated fetch), and Claude-SearchBot (improves search result quality). Similar structure to OpenAI's crawler ecosystem.

Question 4

What are Perplexity's crawlers?

Accepted Answer

Perplexity documents PerplexityBot (surfaces and links websites in Perplexity search results) and Perplexity-User (user-initiated fetch to answer questions). PerplexityBot is the main crawler for visibility.

Question 5

Does Perplexity-User follow robots.txt?

Accepted Answer

Perplexity's documentation notes that since the fetch is user-requested, this fetcher 'generally ignores robots.txt rules.' The user explicitly asked to access the content, so Perplexity honors that request.

Question 6

How do I allow OpenAI's search crawler in robots.txt?

Accepted Answer

Add an allow rule for OAI-SearchBot and verify your robots.txt syntax. OpenAI recommends allowing OAI-SearchBot for inclusion in ChatGPT search results.

Question 7

Should I block AI crawlers?

Accepted Answer

It's a business decision: block training crawlers if you want to reduce training use; allow search crawlers if you want visibility and citations. Many companies separate 'search inclusion' from 'training inclusion' where possible.

Question 8

Can my WAF block AI crawlers even if robots.txt allows them?

Accepted Answer

Yes. If your WAF blocks their IPs or challenges them, they may never fetch your content. Perplexity explicitly provides WAF configuration guidance and IP ranges for verification to prevent accidental blocking.

Question 9

What is llms.txt?

Accepted Answer

/llms.txt is a proposed standard: a markdown file at your site root that provides LLM-friendly guidance and curated links to key resources, intended mainly for inference-time use when an assistant is answering questions about your site.

Question 10

Is llms.txt the same as robots.txt?

Accepted Answer

No: robots.txt provides access control signals for crawlers (allow/disallow). llms.txt provides curated 'how to use this site' context for LLMs/agents. They're complementary standards for different purposes.

Question 11

Should I publish markdown versions of key pages?

Accepted Answer

If feasible, yes. The llms.txt proposal suggests making clean markdown versions available (e.g., .md appended URLs) to reduce parsing friction. This gives AI systems cleaner, more extractable content.

Question 12

Do sitemaps and RSS still matter for GEO?

Accepted Answer

Yes. Sitemaps aid discovery; RSS can help syndication and recency detection. They don't guarantee citation, but they remove preventable barriers to discovery and help AI systems find your content.

Q1Which OpenAI crawlers matter for GEO?

Q2If I block GPTBot, can I still appear in ChatGPT search?

Q3What are Anthropic's crawlers?

Q4What are Perplexity's crawlers?

Q5Does Perplexity-User follow robots.txt?

Q6How do I allow OpenAI's search crawler in robots.txt?

Q7Should I block AI crawlers?

Q8Can my WAF block AI crawlers even if robots.txt allows them?

Q9What is llms.txt?

Q10Is llms.txt the same as robots.txt?

Q11Should I publish markdown versions of key pages?

Q12Do sitemaps and RSS still matter for GEO?

Explore Other Topics