11 min read

Reducing Headless Chrome Memory During Large Crawls

A headless browser crawl that works perfectly against two hundred URLs will frequently die against two hundred thousand, and the failure is almost never in the crawl logic. Each Chromium renderer holds a few hundred megabytes, several of them run at once, and the process does not return everything it takes. This page is part of Configuring Headless Browsers for JS-Heavy Sites and covers keeping a long crawl inside its memory budget.

What each configuration costs per worker A horizontal bar chart of approximate resident memory per worker under four configurations. Launching a fresh browser per URL costs around 620 megabytes. A fresh context per URL on a shared browser costs around 340. A reused context with a fresh page per URL costs around 210. A reused context with request interception blocking images and fonts costs around 150. The gap between the first and last is roughly fourfold. APPROXIMATE RESIDENT MEMORY PER WORKER Fresh browser per URL ~620 MB Fresh context per URL ~340 MB Reused context, fresh page ~210 MB Reused context + interception ~150 MB Figures are indicative and vary by page weight, but the ordering is stable. The single largest saving is not launching a browser per URL, and the second is not downloading bytes you discard.

Environment isolation and dependency declaration 

set -euo pipefail
export CRAWL_CONTAINER_MEM_MB=4096       # the cgroup limit, stated explicitly
export CRAWL_WORKER_BUDGET_MB=300        # measured, not guessed
export CRAWL_RECYCLE_EVERY=200           # pages before a browser restart
export CRAWL_BLOCK_TYPES="image,font,media"
export CRAWL_STREAM_OUTPUT=1             # never accumulate records in memory
# Concurrency is derived, never typed by hand:
export CRAWL_CONCURRENCY=$(( (CRAWL_CONTAINER_MEM_MB - 512) / CRAWL_WORKER_BUDGET_MB ))

Deriving concurrency from the container limit rather than picking a number is the whole discipline in one line. A hand-picked concurrency that worked on a laptop with sixteen gigabytes becomes an OOM kill on a four-gigabyte runner, and the failure arrives partway through a long run rather than at startup where it would be obvious.

Implementation 

#!/usr/bin/env python3
# /opt/audit/crawl/browser_pool.py
# One browser per worker, recycled on a page count, with everything closed.
from __future__ import annotations
import contextlib
import os
from playwright.sync_api import sync_playwright

RECYCLE_EVERY = int(os.environ.get("CRAWL_RECYCLE_EVERY", "200"))
BLOCK = set(os.environ.get("CRAWL_BLOCK_TYPES", "").split(","))

LAUNCH_ARGS = [
    "--disable-gpu",
    "--disable-dev-shm-usage",     # use /tmp rather than a small /dev/shm
    "--disable-extensions",
    "--no-first-run",
    "--js-flags=--max-old-space-size=256",   # cap the JS heap per renderer
]


class BrowserPool:
    def __init__(self, pw):
        self._pw = pw
        self._browser = None
        self._ctx = None
        self._pages_served = 0

    def _launch(self) -> None:
        self._browser = self._pw.chromium.launch(args=LAUNCH_ARGS)
        self._ctx = self._browser.new_context()
        if BLOCK:
            self._ctx.route("**/*", self._filter)
        self._pages_served = 0

    def _filter(self, route, request) -> None:
        if request.resource_type in BLOCK:
            route.abort()          # abort, not fulfil: no bytes are transferred
        else:
            route.continue_()

    @contextlib.contextmanager
    def page(self):
        if self._browser is None or self._pages_served >= RECYCLE_EVERY:
            self.close()
            self._launch()
        pg = self._ctx.new_page()
        try:
            yield pg
        finally:
            pg.close()             # always, even on an exception
            self._pages_served += 1

    def close(self) -> None:
        for obj in (self._ctx, self._browser):
            if obj is not None:
                with contextlib.suppress(Exception):
                    obj.close()
        self._ctx = self._browser = None

The choices that matter:

  • The context is reused and the page is not. A fresh page per URL costs little and guarantees no DOM or listener from the previous URL survives; a fresh context per URL costs a great deal more and buys isolation the crawl usually does not need.
  • page.close() is in a finally block. An exception during navigation is common on a large crawl, and a page leaked on every failure is the most reliable way to run out of memory on exactly the pages that were already problematic.
  • --disable-dev-shm-usage is not optional in a container. The default /dev/shm is 64 MB on most container runtimes, and Chromium crashes rather than degrading when it fills.
  • route.abort() rather than route.fulfil() for blocked types. Fulfilling with an empty response still allocates; aborting means the bytes are never requested.
  • The browser is recycled on a page count. Process-level fragmentation is not released by closing pages or contexts, so a long-lived process grows regardless of how careful the code is.
Four things that accumulate Four stacked accumulations. Pages left open hold their whole DOM and JavaScript heap until closed. Contexts hold cookies, storage and their own cache until closed. The browser process itself accumulates fragmentation that no API releases. And the crawler own result list grows if records are held in memory instead of streamed to disk. Each row names what actually frees it. Open pages DOM plus JS heap, per page close every page in a finally block Open contexts cookies, storage, its own cache close the context, not just the page Process fragmentation nothing in the API releases it recycle the browser every N pages The result list every record held until the end stream to disk, never accumulate The third row is the one that defeats otherwise careful code: a process that closes everything correctly still grows, and only restarting it gives the memory back.

Verification and smoke test 

set -euo pipefail
# 1. Concurrency actually fits the container limit.
python3 -c "
import os
mem = int(os.environ['CRAWL_CONTAINER_MEM_MB']); per = int(os.environ['CRAWL_WORKER_BUDGET_MB'])
c = int(os.environ['CRAWL_CONCURRENCY'])
assert c * per + 512 <= mem, f'{c} workers x {per}MB exceeds {mem}MB'
print(f'PASS: {c} workers x {per}MB fits {mem}MB with headroom')"

# 2. Resident memory is flat across a long run, not climbing.
python3 -m audit.crawl --urls /data/smoke/2000_urls.txt --report-rss   | tee /tmp/rss.log
awk 'NR==1{first=$2} {last=$2} END {
  growth=(last-first)/first*100
  printf "RSS grew %.1f%% over the run
", growth
  exit (growth > 25) }' /tmp/rss.log   && echo "PASS: resident memory is stable across the run"

# 3. Blocked resource types are genuinely not fetched.
grep -c '"resource_type":"image"' /data/smoke/requests.jsonl   | awk '{ if ($1 > 0) { print "FAIL: images were fetched"; exit 1 } }'   && echo "PASS: no blocked resource types fetched"

Expected output is three PASS lines with RSS growth in single digits. The second check is the one that catches a regression: growth above roughly a quarter over two thousand pages means something is being retained, and it is far cheaper to find here than at hour three of a production run.

Failure modes

Three ways a long crawl dies Three rows. An OOM kill partway through means concurrency times per-worker footprint exceeds the container limit. Steadily climbing memory with correct cleanup means process-level fragmentation that only a recycle releases. And a crawl that slows down long before it dies is usually swapping, which looks like a slow origin rather than a memory problem. SYMPTOM ROOT CAUSE FIX The container is OOM-killed partway through, with no error from the crawler Concurrency times footprint exceeds the limit, and the kernel decides before you do Derive concurrency from the limit and the measured per-worker footprint Memory climbs steadily even though pages and contexts are closed Process fragmentation no API call returns it to the operating system Recycle the browser restart it every N pages, mid-crawl, without losing the queue The crawl slows down long before anything is killed It is swapping which reads as a slow origin rather than as memory pressure Alert on RSS, not just OOM a rising RSS is the early signal that a kill is coming Only the first is unmistakable. The other two present as performance problems, and teams routinely spend a day tuning timeouts before checking resident memory.

The recycle interval is too long to help

Recycling every two thousand pages on a process that grows measurably every hundred means the recycle happens after the OOM. Set the interval from the measured growth rate: recycle before resident memory reaches roughly seventy percent of the per-worker budget, not on a round number.

Blocking images changes the metrics being collected

Aborting image requests removes exactly the resources that most often determine largest contentful paint, so a crawl with images blocked cannot produce a comparable LCP. Block them for structural crawls — link graphs, status codes, canonical tags — and never for performance measurement, where the same reasoning as normalizing performance data across device types applies: the configuration is part of the measurement.

Memory is fine and the crawl still slows down

Check whether the container is swapping before tuning anything else. A crawl that slows steadily while resident memory approaches the limit is paging, and every symptom of that looks like a slow origin — rising response times, timeouts on heavy pages, and a rate limiter that keeps backing off in response to latency it is itself causing.

FAQ

Why reuse the browser context instead of creating one per URL?

Because a context carries its own cookie jar, storage and cache, and creating one per URL costs several times what a fresh page costs while buying isolation most crawls do not need. A fresh page per URL is already enough to guarantee that no DOM, listener or timer from the previous page survives. Per-URL contexts are worth the cost only when the crawl genuinely needs isolated storage per page, which in practice means authenticated crawls covering several personas.

Why recycle the browser process if pages and contexts are closed correctly?

Because process-level fragmentation is not released by any API call. A long-lived Chromium process that closes everything properly still grows, slowly and steadily, and nothing in the automation library returns that memory to the operating system. Restarting the process on a page count is the only mechanism that reliably gives it back, which is why the interval should be derived from the observed growth rate rather than picked as a round number.

Does blocking images invalidate the crawl?

It invalidates performance measurement and not much else. Images are frequently the largest contentful paint element, so a crawl with them aborted cannot produce an LCP comparable to one without. For structural work — link graphs, status codes, canonical tags, indexability — they contribute nothing and cost a great deal, so blocking them is straightforwardly correct. The rule is that the configuration is part of the measurement: two crawls with different resource filters are not comparable.