Crawlers & Access
What are Anthropic's crawlers?
Updated January 12, 2026GEO Expert Team
Summary
Anthropic documents ClaudeBot (potential training data collection), Claude-User (user-initiated fetch), and Claude-SearchBot (improves search result quality). Similar structure to OpenAI's crawler ecosystem.
Detailed Answer
Anthropic describes:
1. ClaudeBot
- Purpose: Potential training data collection
- User-agent: ClaudeBot
- Can block for training opt-out
2. Claude-User
- Purpose: User-initiated fetch
- Triggered when user asks Claude to access URL
- Respects explicit user requests
3. Claude-SearchBot
- Purpose: Improves search result quality
- For visibility in Claude search features
- Similar to OAI-SearchBot
Configuration similar to OpenAI:
Allow search bots for visibility, optionally block training bot.
anthropicclaudecrawlersrobots-txt