What Is a Robots.txt File?
A robots.txt file is a small text file at the root of a site, like yoursite.com/robots.txt, that tells crawlers which parts of the site they are and are not allowed to visit. It was built for search engines, but it now also controls whether AI crawlers like GPTBot or Google Extended can read your site at all.
How it works
The file lists rules by crawler name, allowing or disallowing specific paths. A crawler is expected to check this file first and follow whatever rules it lists, though following it is voluntary, not enforced.
Why it matters more than it used to
For years, the file mostly controlled search engine crawling. Now it also decides whether the crawlers that feed AI models, like GPTBot for ChatGPT or Google Extended for Gemini, can access your content in the first place. If they are blocked, a model has no way to learn about you from your own site.
A mistake worth checking for
- Some sites disallow all crawlers by accident during a redesign or migration and never notice
- Some sites intentionally block AI crawlers, sometimes without realizing that also removes them from being cited in AI answers
- The file is easy to check yourself: just visit yoursite.com/robots.txt in a browser
What to do about it
See our guide on how to check if your site blocks AI crawlers for the exact steps to review and fix your file.
See where you stand
Check your own AI visibility free, no signup required.