Legal & Ethics
Should I block training crawlers but allow search crawlers?
Updated January 12, 2026GEO Expert Team
Summary
Many site owners do exactly that when supported. OpenAI explicitly distinguishes GPTBot (training) and OAI-SearchBot (search). Anthropic similarly distinguishes training vs user/search bots. This gives you control over how your content is used.
Detailed Answer
Many site owners do exactly that when supported. OpenAI explicitly distinguishes GPTBot (training) and OAI-SearchBot (search). Anthropic similarly distinguishes training vs user/search bots.
How to implement:
```
# Block training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Allow search
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
```
Business considerations:
- Training use: Your content trains future models
- Search use: Your content appears in answers
- Choose based on your content strategy
Note: Monitor provider documentation for policy changes.
trainingcrawlersopt-outrobots-txt