Why This Matters for Search & AI Visibility
Blanket Disallow rules copied from outdated security advisories often block OAI-SearchBot alongside training scrapers, removing your brand from live ChatGPT Search results.
What Should My robots.txt Look Like for AI Search Engines?
An optimal robots.txt for AI search explicitly allows retrieval agents (Googlebot, OAI-SearchBot, PerplexityBot, ClaudeBot) on public content while blocking private app routes (/admin, /api, /checkout) and referencing both your XML sitemap and /llms.txt.
Live Domain Diagnostic
GoCrawl Technical Advisory
How to Fix It Permanently
Separate live search citation bots (OAI-SearchBot, PerplexityBot) from training-only crawlers if desired, or allow all major AI assistants full access to your public marketing and documentation routes.