Crawlers & Access
Should I block AI crawlers?
Updated January 12, 2026GEO Expert Team
Summary
It's a business decision: block training crawlers if you want to reduce training use; allow search crawlers if you want visibility and citations. Many companies separate 'search inclusion' from 'training inclusion' where possible.
Detailed Answer
It's a business decision:
Block training crawlers if:
- You don't want content used for training
- You have copyright concerns
- Content is premium/paywalled
Allow search crawlers if:
- You want visibility in AI search
- You want citations and attribution
- Brand awareness matters
Recommended approach:
```
# Block training, allow search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
```
Note: Each provider's terms and capabilities may change—monitor documentation.
ai-crawlersrobots-txtstrategyblocking