13 min read

Throttling Crawls to Respect Server Rate Limits

An audit crawl that ignores server capacity produces two failures at once: it degrades the very site it is supposed to measure, and it gets itself rate-limited or blocked, corrupting the resulting health-score data with artificial 429s and timeouts. This page is a focused implementation guide for the Managing Crawl Budget & Rate Limiting workflow — a per-host token bucket that paces requests, reacts to 429 Too Many Requests by honoring the server's own Retry-After value, and bounds retries so a stuck host never stalls the whole crawl.

Per-Host Token Bucket with Retry-After Handling Flow diagram: a queued request checks whether its host bucket has at least one token. If not, it sleeps for the refill deficit and rechecks. If a token is available, it sends the GET request. A 200 response returns control to the crawler. A 429 response triggers parsing of the Retry-After header, which zeroes the bucket and sets a cooldown before the next check. Queued request next URL for host Host bucket: tokens ≥ 1? NO Sleep for deficit / rate recheck after refill YES Send GET request consume one token Response status 200 or 429? 200 Return response to crawl frontier 429 Parse Retry-After header zero bucket, set cooldown

Environment Isolation and Dependency Declaration

Pin the throttle's tuning knobs as environment variables and pin the HTTP client version. A floating requests version can change retry/redirect defaults between crawl runs and make throttling behavior non-reproducible across environments.

# /opt/audit/crawl_throttle — absolute working directory
export CRAWL_RATE_PER_SEC=2.0        # steady-state tokens added per second, per host
export CRAWL_BURST_CAPACITY=4.0      # max tokens a host bucket can hold
export CRAWL_MAX_BACKOFF_SEC=120     # hard ceiling on any parsed Retry-After cooldown
export CRAWL_MAX_RETRIES=3           # bounded retries after a 429 before hard failure
# requirements.txt
requests==2.32.3

set -euo pipefail
cd /opt/audit/crawl_throttle
python3 -m venv .venv
.venv/bin/pip install --quiet -r requirements.txt

CRAWL_RATE_PER_SEC and CRAWL_BURST_CAPACITY are read once at process start and shared across every worker thread inside a single crawl run. Changing them requires a restart, which is deliberate — it keeps the observed request rate for a given crawl run auditable after the fact.

Implementation

The core object is a per-host HostBucket managed by a CrawlThrottle registry that creates buckets lazily as new hosts are discovered by the orchestrating distributed crawls frontier. Refill uses time.monotonic(), never time.time(), so a system clock adjustment (NTP step, DST, manual correction) can never cause a burst of free tokens or a false stall.

#!/usr/bin/env python3
# /opt/audit/crawl_throttle/token_bucket.py
"""
Per-host token-bucket rate limiter for the audit crawl fetcher.
Honors HTTP 429 responses and their Retry-After header by zeroing the
bucket and blocking new requests to that host until the server's own
cooldown elapses.
"""

import os
import time
import threading
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.parse import urlparse

import requests

# ── 1. environment-injected tuning knobs, read once at import time ──────────
RATE = float(os.environ.get("CRAWL_RATE_PER_SEC", "2.0"))        # tokens/sec/host
CAPACITY = float(os.environ.get("CRAWL_BURST_CAPACITY", "4.0"))  # bucket ceiling
MAX_BACKOFF = float(os.environ.get("CRAWL_MAX_BACKOFF_SEC", "120"))
MAX_RETRIES = int(os.environ.get("CRAWL_MAX_RETRIES", "3"))


class HostBucket:
    """One token bucket per registered host. Thread-safe via a single lock."""

    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity            # start full so the first burst is allowed
        self.updated_at = time.monotonic()
        self.blocked_until = 0.0          # monotonic deadline set by a 429
        self.lock = threading.Lock()

    def _refill(self) -> None:
        # ── 2. add tokens proportional to elapsed monotonic time ────────────
        now = time.monotonic()
        elapsed = now - self.updated_at
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.updated_at = now

    def acquire(self) -> None:
        # ── 3. block until a hard 429 cooldown clears, then until a token exists ──
        with self.lock:
            now = time.monotonic()
            if now < self.blocked_until:
                time.sleep(self.blocked_until - now)
            self._refill()
            while self.tokens < 1.0:
                deficit = 1.0 - self.tokens
                time.sleep(deficit / self.rate)
                self._refill()
            self.tokens -= 1.0

    def apply_retry_after(self, header_value: str | None) -> float:
        # ── 4. on 429, zero the bucket and set a hard, capped cooldown ──────
        cooldown = min(_parse_retry_after(header_value), MAX_BACKOFF)
        with self.lock:
            self.tokens = 0.0
            self.blocked_until = max(self.blocked_until, time.monotonic() + cooldown)
        return cooldown


def _parse_retry_after(value: str | None) -> float:
    """Retry-After is delta-seconds ('30') or an HTTP-date. Default 30s if unparsable."""
    if not value:
        return 30.0
    value = value.strip()
    if value.isdigit():
        return float(value)
    try:
        target = parsedate_to_datetime(value)
        if target.tzinfo is None:
            target = target.replace(tzinfo=timezone.utc)
        return max((target - datetime.now(timezone.utc)).total_seconds(), 0.0)
    except (TypeError, ValueError):
        return 30.0


class CrawlThrottle:
    """Owns one HostBucket per host, created lazily on first request to that host."""

    def __init__(self, rate: float = RATE, capacity: float = CAPACITY):
        self._rate, self._capacity = rate, capacity
        self._buckets: dict[str, HostBucket] = {}
        self._registry_lock = threading.Lock()

    def _bucket_for(self, host: str) -> HostBucket:
        # ── 5. per-host bucket registry, not one shared bucket for all hosts ─
        with self._registry_lock:
            if host not in self._buckets:
                self._buckets[host] = HostBucket(self._rate, self._capacity)
            return self._buckets[host]

    def fetch(self, url: str, session: requests.Session) -> requests.Response:
        host = urlparse(url).netloc
        bucket = self._bucket_for(host)

        for attempt in range(1, MAX_RETRIES + 1):
            bucket.acquire()                       # ── 6. pace before every attempt
            resp = session.get(url, timeout=10)

            if resp.status_code != 429:
                return resp

            cooldown = bucket.apply_retry_after(resp.headers.get("Retry-After"))
            print(f"429 from {host}: cooling down {cooldown:.1f}s (attempt {attempt})")

            if attempt == MAX_RETRIES:
                resp.raise_for_status()            # ── 7. surface, don't retry forever

        raise RuntimeError(f"unreachable: exhausted retries for {url}")

Line-by-line, the design rests on four decisions. First, HostBucket is keyed per host inside CrawlThrottle._buckets, so a slow-to-respond origin never borrows capacity from — or steals capacity from — any other host in the same crawl. Second, _refill uses time.monotonic() exclusively, which makes the bucket immune to wall-clock jumps. Third, apply_retry_after treats a 429 as authoritative: it zeroes tokens to 0.0 rather than merely delaying the next refill tick, so a burst of already-queued requests to that host cannot slip through during the cooldown window. Fourth, fetch bounds retries with CRAWL_MAX_RETRIES and calls raise_for_status() on the final attempt — a rate-limited host degrades the crawl's completeness for that host, not the entire pipeline's runtime.

Verification and Smoke Test

Run this against a mock server that returns 429 with a known Retry-After value on the first request to a host, then 200 afterward. This is the same acceptance pattern used for the handling dynamic content render pipeline's smoke tests — deterministic input, asserted timing.

set -euo pipefail
cd /opt/audit/crawl_throttle

# 1. start a disposable mock server that 429s once per host, then 200s
.venv/bin/python tests/mock_429_server.py --port 8089 --retry-after 3 &
MOCK_PID=$!
trap 'kill $MOCK_PID' EXIT

# 2. run the throttle against it and capture wall-clock timing
START=$(date +%s)
.venv/bin/python -c "
import requests
from token_bucket import CrawlThrottle
t = CrawlThrottle(rate=2.0, capacity=4.0)
s = requests.Session()
resp = t.fetch('http://127.0.0.1:8089/page', s)
assert resp.status_code == 200, f'expected 200, got {resp.status_code}'
print('PASS: request succeeded after cooldown')
"
END=$(date +%s)
ELAPSED=$((END - START))

# 3. confirm the observed wait matches the configured Retry-After (±1s)
[ "$ELAPSED" -ge 3 ] && [ "$ELAPSED" -le 5 ] \
  && echo "PASS: cooldown honored (${ELAPSED}s elapsed)" \
  || echo "FAIL: expected ~3s cooldown, observed ${ELAPSED}s"

Expected output:

429 from 127.0.0.1:8089: cooling down 3.0s (attempt 1)
PASS: request succeeded after cooldown
PASS: cooldown honored (3s elapsed)

If the elapsed time is near zero, the bucket is not actually blocking on blocked_until — check that apply_retry_after and acquire share the same HostBucket instance rather than two separate buckets keyed slightly differently (a trailing port number or www. prefix mismatch is the usual cause).

Failure Modes

A single global bucket instead of one per host. Under load, this either starves a fast-responding host of its fair share because a slow host keeps draining shared tokens, or lets a burst against one host proceed while other hosts sit idle. Diagnosis and fix both live in the registry, not the bucket:

# diagnostic: log bucket key alongside host on every acquire()
.venv/bin/python -c "
from token_bucket import CrawlThrottle
t = CrawlThrottle()
print(sorted(t._buckets.keys()))  # should be one entry per distinct netloc, never empty/shared
"

Clock skew corrupts the parsed Retry-After cooldown. If the crawl host's clock drifts from the origin's, an HTTP-date Retry-After value computes a delta that is too short (under-throttles, gets rate-limited again immediately) or absurdly long (stalls that host for the rest of the crawl window).

# diagnostic: compare crawl host clock to a trusted time source
timedatectl status | grep -E 'System clock synchronized|NTP service'
# fix: force an immediate sync before a scheduled crawl run
sudo chronyc makestep

Retrying forever instead of surfacing a hard failure. An unbounded retry loop against a host that is permanently rate-limiting (misconfigured WAF rule, IP-level block) will not error — it will simply run until the crawl's overall timeout kills the whole job, silently dropping every other host queued behind it.

# diagnostic: count consecutive 429s per host in the crawl log
grep '429 from' /var/log/audit/crawl.log | sort | uniq -c | sort -rn | head
# fix: lower CRAWL_MAX_RETRIES and let the crawl continue past that host
export CRAWL_MAX_RETRIES=2

FAQ

Should every host get its own bucket, or is one global bucket enough?

Per-host, always. A single global bucket lets one slow or rate-limit-happy host consume the entire crawl's request allowance and starve every other host in the frontier, or lets a fast host burst past what a single origin can tolerate. Keyed buckets — one per registered host — are the only configuration that scales correctly once a crawl spans more than one domain, which is also why orchestrating distributed crawls across workers shards its frontier by host rather than by raw URL count.

What Retry-After formats do real servers send, and do I need to handle both?

Yes. RFC 9110 allows Retry-After to be either an integer number of delta-seconds (Retry-After: 30) or a full HTTP-date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). CDNs and load balancers commonly send the date form. A parser that only handles the integer case throws on the date form and silently falls back to a default cooldown, which under-throttles the exact host that just asked to be throttled harder.

What happens if the crawler's clock is skewed from the origin server's clock?

An HTTP-date Retry-After is computed relative to the server's clock, so a skewed crawl host produces a cooldown that is too short or too long. Sync the crawl host with NTP, cap the parsed cooldown at a sane maximum such as 120 seconds, and never let a negative delta — server clock ahead of the crawler — result in a zero-wait retry loop.

Is honoring 429 responses enough, or do I still need a fixed rate cap?

You need both. Many origins never return 429 at all — they just get slow, time out, or start serving 5xx errors under load, with no signal telling the crawler to back off. A steady-state token-bucket rate cap keeps the crawl polite even against servers that give no rate-limit signaling; Retry-After handling is the fast-reaction layer for the servers that do, and recovering gracefully when budget is already exhausted is covered separately in recovering from crawl budget exhaustion.