How to Check If Your Site Blocks AI Crawlers

Start with robots.txt

Visit yoursite.com/robots.txt directly in a browser. Look for a block that names a specific AI crawler, for example User-agent: GPTBot, followed by Disallow: /. That means the site is telling that bot to stay out, either entirely or from specific paths.

The crawlers worth checking for

• GPTBot, used by OpenAI to train ChatGPT
• OAI-SearchBot, OpenAI's separate crawler for live search results, distinct from GPTBot
• ClaudeBot, used by Anthropic
• Google-Extended, which controls whether Google's AI features can use your content
• PerplexityBot and Perplexity-User, used by Perplexity
• Applebot-Extended, used for Apple's AI features

Check hosting level blocking too

robots.txt is a request, not a lock. It relies on the crawler choosing to respect it. Separately, some hosting platforms and CDNs, including some Vercel and Cloudflare settings, offer a toggle to block AI bots outright at the network level, which can be turned on by default or by someone on your team without much thought. This will not show up in robots.txt at all, so it needs a separate check with your host or CDN provider.

Why this matters more than it seems

A blocked crawler means an AI model has no way to read your current content, which means it is working from stale training data at best, or nothing at all. This is often the single biggest, most fixable reason a company shows up nowhere in AI answers about its own category.

Run a full check

QueryTrace's free AI readiness audit checks crawler access, your llms.txt file, and a few other AI discoverability signals for any domain in one pass.