What Is Crawl Budget?
Crawl budget is the limited number of pages a crawler is willing to visit on your site within a given period, based on your site's size, how fast it responds, and how important the crawler judges it to be. Once a crawler's budget for your site runs out, it stops, whether or not it has seen everything.
Why crawlers ration their attention
The web is enormous, and no crawler, whether it belongs to Google or an AI company, can afford to fully re read every page on every site constantly. Instead it budgets a certain number of requests per site, spending more time on sites that update often, respond quickly, and seem worth the effort.
What wastes crawl budget
- A slow server that takes too long to respond to each request
- Huge numbers of low value, near duplicate pages that eat up requests without adding real information
- Broken links and redirect chains that send a crawler in circles
Why this matters for AI crawlers specifically
AI crawlers like GPTBot and ClaudeBot face the same constraint, and companies with a huge number of pages have more reason to worry about it than a small site. If your best, most important pages are buried behind thousands of thin ones, a crawler with limited budget may never reach them.
What actually helps
Keep your site fast, remove or consolidate low value pages, fix broken links, and make sure your most important pages are easy to reach in just a click or two from the homepage. An llms.txt file also helps here, since it points a model straight at what matters instead of relying on it to discover that through crawling alone.
See where you stand
Check your own AI visibility free, no signup required.