13 min read

Recovering from Crawl Budget Exhaustion

A crawl that spends most of its budget on faceted navigation, stale session-ID URLs, or an infinite calendar widget never reaches the pages that actually need re-indexing. This is one of the most common findings a remediation runbook has to close, and it is part of Writing Remediation Playbooks for Audit Failures: a repeatable procedure that turns a vague "crawl coverage looks low" alert into a ranked list of URL patterns to cut, and a verified confirmation that the cut worked. The underlying constraint is described in more depth in managing crawl budget and rate limiting, but the short version is that every site gets a finite number of requests per crawl window, and every request spent on a duplicate facet combination is a request not spent on a page you actually want indexed.

Crawl Budget Recovery Pipeline Five-stage diagram showing raw access logs flowing into a rank-and-classify step that buckets requests into facet, session, and calendar patterns, then into exclusion-rule generation, deployment and bot-compliance check, and finally a re-crawl step that confirms reclaimed budget. Access Logs bot requests only gzip + rotated Rank & Classify awk pattern buckets facet / session / cal Exclusion Rules robots.txt disallow or rel=canonical Deploy confirm bot compliance 1–3 day lag Re-crawl verify hit-share shift budget reclaimed

Where Crawl Budget Actually Goes

Before writing any exclusion rule, find out exactly which URL shapes are consuming requests. Three patterns account for the overwhelming majority of wasted crawl budget on mid-size and large sites. Faceted navigation — color, size, material, and sort-order parameters layered on top of a category page — multiplies one canonical listing into hundreds of near-duplicate combinations, each of which the crawler treats as a distinct URL until told otherwise. Session-ID parameters appended by legacy application servers create a functionally infinite set of URLs, since every crawl session mints new IDs that are never reused. Infinite or auto-generated calendar widgets are the worst offender in practice: a "next month" link that resolves cleanly for any date range means a crawler can walk forward for years without ever hitting a dead end, burning requests on empty date pages decades into the future.

All three share a signature: they are cheap to generate on the server and expensive to crawl, and none of them show up clearly in a dashboard until you go straight to the source — the access log lines the crawler itself produced. That is the diagnostic step this playbook automates, before moving to the fix.

Ranking Budget-Wasting Paths from Access Logs

Run this against a representative window of access logs — at least seven days, so weekly patterns like calendar navigation don't get missed. The script isolates crawler requests, strips and classifies the query string into the three known waste patterns, counts hits per pattern, and emits a ranked candidate list for exclusion in one pass.

#!/usr/bin/env bash
# /opt/audit/crawl-budget/rank_wasted_paths.sh
# Rank crawl-budget-wasting URL patterns from access logs.
set -euo pipefail

LOG_GLOB="/var/log/nginx/access.log*"          # current + rotated, gzip or plain
BOT_UA_PATTERN='Googlebot|bingbot|DuckDuckBot'  # crawlers you care about
MIN_HITS=${MIN_HITS:-25}                        # ignore noise below this count
TOP_N=${TOP_N:-50}

# 1. pull only crawler-attributed request lines across rotated and live logs
zgrep -hE "$BOT_UA_PATTERN" $LOG_GLOB \
  | awk -v min_hits="$MIN_HITS" '
    {
      # 2. isolate the request path between the request-line quotes
      match($0, /"[A-Z]+ [^ ]+ HTTP/);
      req = substr($0, RSTART, RLENGTH);
      split(req, parts, " ");
      path = parts[2];

      # 3. split base path from query string so patterns bucket correctly
      qpos = index(path, "?");
      base  = (qpos > 0) ? substr(path, 1, qpos - 1) : path;
      query = (qpos > 0) ? substr(path, qpos + 1)     : "";

      # 4. classify into the three known crawl-budget-waste shapes
      pattern = base;
      if (query ~ /(^|&)(sid|sess|tok|sessionid|phpsessid|jsessionid)=/)
        pattern = base "?[session-id]";
      else if (query ~ /(^|&)(color|size|material|sort|filter)=/)
        pattern = base "?[facet]";
      else if (base ~ /\/calendar\/[0-9]{4}\/[0-9]{2}\//)
        pattern = "/calendar/[YYYY]/[MM]/";
      else if (query != "")
        pattern = base "?[other-query]";

      count[pattern]++;
    }
    END {
      # 5. emit hit count + pattern, filtered to patterns above the noise floor
      for (p in count) {
        if (count[p] >= min_hits) print count[p], p;
      }
    }' \
  | sort -rn \
  | head -n "$TOP_N"

Line by line: the zgrep -hE call reads gzip and plain logs identically and drops filenames from the output with -h, keeping only lines whose user agent matches a known crawler so the ranking reflects crawl traffic and not real visitor sessions. Inside the awk block, the match()/substr()/split() sequence pulls the path out of the quoted request line without depending on a fixed column count, which matters because referrer and user-agent fields in combined log format can contain spaces. The query string is then separated from the base path so two different session-ID values on the same page count as one pattern instead of two rows. The four-way classification is deliberately ordered — session IDs are checked first because a session parameter can co-occur with a facet parameter, and the session waste is usually the larger problem. Anything left over that still carries a query string falls into an [other-query] bucket, which is where you go hunting for patterns the script doesn't know about yet. The MIN_HITS floor keeps one-off crawl anomalies out of the ranked list so you're only looking at patterns with a real, recurring cost.

A typical run against a mid-size e-commerce log surfaces output like this:

14822 /catalog/mens-shoes?[facet]
 6190 /calendar/[YYYY]/[MM]/
 3401 /account/dashboard?[session-id]
  812 /catalog/mens-shoes
  204 /blog/care-guide

The first three rows account for over 80% of crawl hits in this window while the two legitimate content pages at the bottom barely register — a textbook budget-exhaustion signature.

Turning the Ranking into Exclusion Rules

Once a pattern clears the noise floor and you've manually confirmed a handful of sample URLs, decide between two remediation paths, and this is the step where the remediation playbook pattern of pre-check, fix, and post-verify actually pays off. Patterns that should never be crawled at all — the session-ID and infinite-calendar rows above — get a robots.txt disallow rule:

User-agent: *
Disallow: /account/dashboard*sid=*
Disallow: /calendar/2028/
Disallow: /calendar/2029/

Facet patterns are different: real visitors use them, and some facet URLs may already carry inbound links or ranking signal, so blocking them outright can erase equity you'd rather consolidate. For those, add or confirm a rel="canonical" tag pointing at the unfiltered category page instead of a disallow rule, so the crawler can still reach the page, read the canonical signal, and fold the duplicate into the parent URL rather than treating it as a wall.

Verification

Confirm the fix in two stages: first that the rule is live and syntactically valid, then — after a full crawl cycle — that the crawler actually changed behavior.

set -euo pipefail

# 1. confirm the new disallow lines are live and well-formed
curl -fsS https://example.com/robots.txt | grep -E "sid=|/calendar/202[89]/"

# 2. re-run the ranking script after one full crawl cycle (7+ days later)
MIN_HITS=25 /opt/audit/crawl-budget/rank_wasted_paths.sh > after.txt
diff <(sort -k2 before.txt) <(sort -k2 after.txt) || true

# 3. confirm hit share has shifted toward indexable content
awk '{sum+=$1} END{print "total bot hits:", sum}' after.txt

Expect the excluded patterns to drop out of the ranked list entirely, or fall well below their prior hit counts within the first re-crawl window, with the freed requests showing up as increased hits against real content paths further down the list.

Failure Modes

A disallow rule blocks a facet URL that was already indexed and ranking

Blocking with robots.txt prevents the crawler from ever seeing a canonical tag on that page, so instead of consolidating into the parent URL it can sit in the index as an orphaned, un-refreshable result.

# diagnostic: check whether the blocked pattern still has indexed pages
curl -fsS "https://www.google.com/search?q=site:example.com+inurl:sid="
# fix: replace the disallow line with a canonical tag on the source template,
# then request removal of the disallow rule so the crawler can read the tag

The session-ID regex misses a vendor-specific parameter name

If the ranking script's pattern list doesn't include the parameter your application actually uses, those requests fall into the generic [other-query] bucket and never get flagged with enough specificity to act on.

# diagnostic: list the top query-string parameter names by frequency
zgrep -hE 'Googlebot|bingbot' /var/log/nginx/access.log* \
  | grep -oE '\?[^"& ]*' | tr '&' '\n' | cut -d= -f1 | sort | uniq -c | sort -rn | head -20
# fix: add the confirmed parameter name to the session-id branch of the awk script

Excluding the calendar pattern also blocks legitimate near-term dates

A disallow rule scoped only by year can accidentally cut off the current and next month if the pattern boundary is set too aggressively, which removes a page real users still navigate to. Scope the exclusion to a rolling window — for example, disallow years beyond the current year plus one — rather than a hardcoded cutoff, and revisit the rule annually. Throttling the crawler itself, as covered in throttling crawls to respect server rate limits, reduces the damage of a miscalibrated exclusion but does not replace fixing the pattern boundary.

FAQ

How do I tell a facet parameter apart from a legitimate unique page in the logs?

Group requests by path plus parameter name, ignoring the value, and compare the distinct-value count against total hits. A facet parameter like color or sort produces dozens of values on the same base path with near-identical response sizes, while a legitimate page has one URL per piece of content. If a single base path plus parameter name accounts for more than a few percent of total bot hits, treat it as a facet candidate and confirm by sampling five URLs manually before excluding.

What if my session ID parameter is not named sessionid or PHPSESSID?

The classification regex only knows the parameter names you give it, so audit a sample of query strings first with a frequency count of parameter keys across the log window. Common non-standard names include sid, sess, tok, and vendor-specific values like awselb or visid. Add each confirmed name to the pattern list in the ranking script and re-run it before generating any exclusion rules.

Should I use a robots.txt disallow or a canonical tag to reclaim budget?

Use a robots.txt disallow for patterns that were never meant to be crawled at all, such as session-ID variants and infinite calendar pagination beyond a reasonable date range — it stops the fetch entirely. Use a canonical tag instead for facet URLs that are still useful to real visitors and might carry backlinks, since disallowing them blocks the crawler from ever seeing the canonical signal on the page itself.

How long after excluding these paths should I expect crawl budget to recover?

Expect the crawler to honor a fresh robots.txt within one to three days for most search engine bots, and expect the hit-share shift toward indexable pages to become visible in your own access logs over one to two full crawl cycles. Recovery is gradual because crawlers keep a queue of previously discovered URLs and drain it rather than dropping disallowed paths instantly.