My content is blocked—what should I check?
Summary
Check robots.txt rules, meta robots noindex/nofollow tags, WAF rules and bot challenges, logged-out accessibility, and server-side rendering for critical content. Any of these can prevent AI access.
Detailed Answer
1. robots.txt:
- Check for Disallow rules
- Verify AI bot user-agents allowed
- Test with robots.txt validator
2. Meta robots:
- Check for noindex tag
- Check for nofollow tag
- Verify in page source
3. WAF/Firewall:
- Review bot management rules
- Check for AI crawler blocks
- Test from different IPs
4. Authentication:
- Test logged-out access
- Check for login walls
- Verify paywall configuration
5. Rendering:
- Test with JavaScript disabled
- Check SSR implementation
- Verify content in initial HTML
Tools:
- Browser dev tools (Network tab)
- Fetch as bot tools
- Log analysis