Crawl budget is one of those terms that sounds technical enough to end a conversation. It is worth understanding properly, because the misunderstandings are expensive.
Myth 1: "We need to increase our crawl budget"
You cannot request more crawl budget. There is no setting. Search engines allocate fetching based on how much they think your site is worth crawling and how fast it responds.
What you can do is make the crawl you get count:
- Remove URLs that should not exist. Faceted navigation, tracking parameters, internal search results, paginated archives nobody reads.
- Return honest status codes. A 404 for a page that is gone. Not a redirect to the homepage.
- Keep important pages shallow. Three clicks, not nine.
- Serve fast, stable responses. A site that times out gets crawled less.
The framing matters. "Increase crawl budget" leads to bad decisions. "Stop wasting crawl" leads to good ones.
Myth 2: "robots.txt will remove the page from Google"
robots.txt stops a crawler fetching a URL. It does not remove it from the index. A blocked URL can still appear in results, based on links and anchor text, with no description.
If you want a page out of the index, you need one of:
- a noindex meta tag or X-Robots-Tag header, on a page the crawler can fetch,
- a 404 or 410 if the page is gone,
- the URL Removal tool for emergencies.
And never do both: blocking the fetch prevents the crawler from ever seeing the noindex.
Myth 3: "Sitemaps tell Google what to crawl"
A sitemap is a suggestion list, not a queue. It helps discovery, particularly for new or poorly linked pages. It does not grant priority.
A sitemap full of non-canonical, redirected or noindexed URLs actively wastes attention. Keep it to 200-status, canonical, indexable pages, and maintain the lastmod dates honestly.
The actual diagnostic
If pages are not being indexed, ask in this order:
- Is it linked? From a page that is itself indexed, with a crawlable anchor.
- Is it allowed? robots.txt, meta robots, X-Robots-Tag, canonical.
- Does it return 200? Check for soft 404s: a 200 response with "not found" content.
- Is it unique? Near-duplicates get consolidated, and the wrong one sometimes survives.
- Is it worth indexing? Thin pages with no unique value get crawled and dropped, correctly.
Only after those five does crawl budget become a plausible explanation. In most audits, it is number one or two, not five.
A quick test
Take a page you expect to rank and check three things in Search Console: whether it is indexed, when it was last crawled, and which page links to it. If it is not indexed, the reason is almost always in that list, and it is almost never the crawl budget.