12 min read

Automating Screaming Frog with Python Scripts

Running Screaming Frog manually through its GUI is incompatible with any repeatable audit pipeline: there is no scheduled execution, no structured output contract, and no way to react programmatically to rate-limit signals or schema drift. This page shows how to replace that manual step with a Python subprocess controller that generates per-domain configurations, drives the headless Screaming Frog CLI from a locked-down container, streams crawl output for real-time anomaly detection, and pushes normalised Parquet artifacts to cloud storage — all wired into the broader automated crawling pipeline.

Pipeline overview

The diagram below shows the four stages the controller orchestrates, from container boot to artifact upload.

Screaming Frog Python automation pipeline stages Four boxes connected by arrows showing: 1 Runtime isolation (Docker + licence), 2 Dynamic config (.seospider), 3 Subprocess execution (--headless), 4 Normalise and upload (Parquet to S3) 1 · Runtime isolation 2 · Dynamic config gen 3 · Subprocess execution 4 · Normalise & upload

Environment isolation and dependency declaration

Pin every dependency so the crawl is reproducible across CI runners, local machines, and scheduled jobs. Build a minimal Docker image from the Eclipse Temurin JRE (OpenJDK 17) — do not use a JDK image in production; the JRE is ~200 MB lighter. Download the Screaming Frog Linux CLI binary at image build time and inject SF_LICENSE_KEY at runtime via an environment variable, never baked into the image layer.

# Dockerfile — pin both the JRE tag and the Screaming Frog version
FROM eclipse-temurin:17.0.11_9-jre-alpine

RUN apk add --no-cache curl unzip bash

ARG SF_VERSION="21.4"
RUN curl -fsSL \
      https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider-${SF_VERSION}.zip \
      -o /tmp/sf.zip && \
    unzip /tmp/sf.zip -d /opt/screamingfrog && \
    chmod +x /opt/screamingfrog/ScreamingFrogSEOSpider && \
    rm /tmp/sf.zip

# Licence key injected at runtime — never embed in the image
ENV SF_LICENSE_KEY=""

RUN adduser -D -s /bin/sh crawler
USER crawler
WORKDIR /data

ENTRYPOINT ["/opt/screamingfrog/ScreamingFrogSEOSpider"]

Python-side requirements (requirements.txt, pinned):

pandas==2.2.2
pyarrow==16.0.0
pydantic==2.7.1
boto3==1.34.101

Export these shell variables before running the controller locally or in CI:

export SF_LICENSE_KEY="your-licence-key"
export CRAWL_OUTPUT_DIR="/data/crawls/$(date +%Y%m%d)"
export S3_BUCKET="your-audit-artifacts-bucket"
export S3_PREFIX="screaming-frog"

Implementation

Stage 1 — Headless subprocess controller

The controller wraps the Screaming Frog CLI in subprocess.run. The key flags are --headless (suppresses the AWT display requirement) and --config (points to the per-domain config generated in stage 2). Pass check=True so any non-zero exit code surfaces immediately as a CalledProcessError rather than silently producing an empty artifact.

# crawler/controller.py
import subprocess
import os
from pathlib import Path


def run_headless_crawl(
    url: str,
    config_path: str,
    output_dir: str,
    timeout_seconds: int = 3600,
) -> subprocess.CompletedProcess:
    """
    Launch ScreamingFrogSEOSpider in headless mode.
    Raises subprocess.CalledProcessError on non-zero exit.
    Raises subprocess.TimeoutExpired if the crawl exceeds timeout_seconds.
    """
    Path(output_dir).mkdir(parents=True, exist_ok=True)

    env = os.environ.copy()
    # Guard: abort if licence key is absent rather than silently running in demo mode
    if not env.get("SF_LICENSE_KEY"):
        raise EnvironmentError("SF_LICENSE_KEY is not set — crawl aborted.")

    cmd = [
        "ScreamingFrogSEOSpider",
        "--headless",
        "--url", url,
        "--config", os.path.abspath(config_path),   # absolute path required in CI
        "--save-crawl",
        "--output-folder", os.path.abspath(output_dir),
        "--export-tabs", "Internal:All",
    ]

    return subprocess.run(
        cmd,
        env=env,
        timeout=timeout_seconds,
        check=True,
        capture_output=True,
        text=True,
    )

Stage 2 — Dynamic config generation

Static .seospider profiles cannot adapt to per-domain rate limits, JavaScript rendering requirements, or crawl scope changes. Instead, generate a fresh config before each crawl run. When the target is a JavaScript-rendered SPA, set RenderMode = JavaScript — the same rendering decision described in the headless browser configuration guide. For rate-sensitive domains, keep MaxThreads at 5 and lower RateLimit before adding threads; reversing that order triggers WAF blocks before the crawl reaches a meaningful URL sample.

# crawler/config_gen.py
import configparser
import os
from pathlib import Path


def generate_crawl_config(
    target_domain: str,
    output_dir: str,
    rate_limit_per_second: int = 2,
    max_threads: int = 5,
    js_render: bool = False,
) -> str:
    """
    Write a .seospider config to a temp path and return the absolute path.
    Overwrites any existing config for this domain to avoid stale settings.
    """
    config = configparser.ConfigParser()

    config["Crawl"] = {
        "StartUrl": f"https://{target_domain}",
        "MaxThreads": str(max_threads),
        "RateLimit": str(rate_limit_per_second),
        "MaxUrls": "0",            # 0 = unlimited
        "FollowInternalNofollow": "false",
    }

    config["Spider"] = {
        "CheckSitemapLinks": "true",
        "CrawlLinkedXMLSitemaps": "true",
    }

    if js_render:
        config["Rendering"] = {
            "RenderMode": "JavaScript",
            "WaitForNetworkRequests": "true",
            "NetworkRequestTimeout": "5000",
        }

    # Write to a domain-scoped temp path, not /tmp/dynamic.seospider,
    # so parallel crawl jobs do not overwrite each other's configs
    config_path = os.path.join(output_dir, f"{target_domain}.seospider")
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    with open(config_path, "w") as fh:
        config.write(fh)

    return os.path.abspath(config_path)

Stage 3 — Real-time WAF detection via streaming

When managing crawl budget and rate limiting matters, polling stdout line-by-line lets you terminate the crawl the moment a WAF throttle signal appears rather than waiting for the full timeout to expire.

# crawler/stream_runner.py
import subprocess
import signal
import sys
import os
from pathlib import Path


WAF_SIGNALS = ["Rate limit exceeded", "429 Too Many Requests", "503 Service Unavailable"]


def run_with_streaming(
    url: str,
    config_path: str,
    output_dir: str,
    timeout_seconds: int = 3600,
) -> int:
    """
    Run the crawl with real-time stdout monitoring.
    Returns the process exit code.
    Sends SIGTERM and returns 1 on WAF detection.
    """
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    env = os.environ.copy()

    cmd = [
        "ScreamingFrogSEOSpider",
        "--headless",
        "--url", url,
        "--config", os.path.abspath(config_path),
        "--save-crawl",
        "--output-folder", os.path.abspath(output_dir),
        "--export-tabs", "Internal:All",
    ]

    with subprocess.Popen(
        cmd,
        env=env,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    ) as proc:
        try:
            for line in proc.stdout:
                sys.stdout.write(line)

                if any(signal_str in line for signal_str in WAF_SIGNALS):
                    sys.stderr.write(f"[ALERT] WAF signal detected — terminating crawl: {line.strip()}\n")
                    proc.send_signal(signal.SIGTERM)
                    return 1

            proc.wait(timeout=timeout_seconds)
        except subprocess.TimeoutExpired:
            proc.kill()
            raise

    return proc.returncode


def _graceful_exit(signum, frame):
    print("[INFO] Received termination signal — flushing and exiting.", flush=True)
    sys.exit(0)


signal.signal(signal.SIGINT, _graceful_exit)
signal.signal(signal.SIGTERM, _graceful_exit)

Stage 4 — CSV normalisation and Parquet upload

Raw Screaming Frog exports carry two reliable pitfalls: UTF-8 BOM on the first byte (corrupting the first column name) and column ordering that changes between Spider versions. The normalisation step strips the BOM with encoding='utf-8-sig', validates each row against a Pydantic model, serialises to Parquet for efficient downstream querying, and pushes the artifact to S3. This is the handoff point for storing and versioning crawl artifacts.

# crawler/normalise.py
import pandas as pd
import boto3
import io
import os
from pydantic import BaseModel, Field, ValidationError
from typing import Optional
from botocore.exceptions import ClientError


class CrawlRecord(BaseModel, extra="forbid"):
    address: str = Field(alias="Address")
    status_code: int = Field(alias="Status Code")
    title: Optional[str] = Field(alias="Title 1", default=None)
    indexability: str = Field(alias="Indexability")
    content_type: Optional[str] = Field(alias="Content Type", default=None)
    word_count: Optional[int] = Field(alias="Word Count", default=None)


def normalise_export(csv_path: str) -> pd.DataFrame:
    """
    Read a Screaming Frog 'Internal:All' CSV, strip BOM, enforce types,
    validate with Pydantic, and return a clean DataFrame.
    Raises ValueError listing the first invalid row if schema validation fails.
    """
    df = pd.read_csv(
        csv_path,
        encoding="utf-8-sig",     # strips UTF-8 BOM — use 'utf-8' and you get a corrupted header
        dtype={"Status Code": "Int64", "Word Count": "Int64"},
    )
    df.columns = df.columns.str.strip()

    records = []
    errors = []
    for idx, row in df.iterrows():
        try:
            records.append(CrawlRecord(**row.to_dict()).model_dump())
        except ValidationError as exc:
            errors.append(f"Row {idx}: {exc.error_count()} validation error(s)")

    if errors:
        raise ValueError(f"Schema validation failed on {len(errors)} rows:\n" + "\n".join(errors[:5]))

    return pd.DataFrame(records)


def upload_parquet(df: pd.DataFrame, bucket: str, key: str) -> None:
    """Serialise df to Parquet in memory and push to S3."""
    buf = io.BytesIO()
    df.to_parquet(buf, engine="pyarrow", index=False, compression="snappy")
    buf.seek(0)

    client = boto3.client("s3")
    try:
        client.put_object(Bucket=bucket, Key=key, Body=buf.getvalue())
    except ClientError as exc:
        raise RuntimeError(
            f"S3 upload to s3://{bucket}/{key} failed: "
            f"{exc.response['Error']['Message']}"
        ) from exc

Verification and smoke test

After a first run, confirm the pipeline succeeded at each stage:

# 1 — container health: licence validated, no X11 errors
docker run --rm \
  -e SF_LICENSE_KEY="${SF_LICENSE_KEY}" \
  your-sf-image:21.4 --version
# Expected: ScreamingFrogSEOSpider 21.4 ... License: Valid (CLI Mode)

# 2 — config present and non-empty
test -s "${CRAWL_OUTPUT_DIR}/example.com.seospider" && echo "Config OK"

# 3 — crawl artifacts exist
ls -lh "${CRAWL_OUTPUT_DIR}/"
# Expected: internal_all.csv  crawl.seospider  cli.log

# 4 — row count sanity check (replace 500 with your expected minimum)
python3 - <<'EOF'
import pandas as pd, sys
df = pd.read_csv("/data/crawls/$(date +%Y%m%d)/internal_all.csv", encoding="utf-8-sig")
assert len(df) >= 500, f"Only {len(df)} rows — crawl likely incomplete"
assert df["Address"].notna().all(), "Null addresses detected"
assert df["Status Code"].notna().all(), "Missing status codes"
print(f"OK — {len(df)} URLs with clean schema")
EOF

# 5 — Parquet artifact in S3
aws s3 ls "s3://${S3_BUCKET}/${S3_PREFIX}/" | grep ".parquet"

Expected output from step 3 after a complete crawl:

-rw-r--r-- 1 crawler crawler 2.4M Jun 21 09:42 internal_all.csv
-rw-r--r-- 1 crawler crawler  12K Jun 21 09:42 crawl.seospider
-rw-r--r-- 1 crawler crawler  88K Jun 21 09:43 cli.log

Failure signal: If cli.log contains java.awt.HeadlessException, the --headless flag was dropped or overridden. If it contains License: Demo Mode, SF_LICENSE_KEY was not injected into the container environment.

Failure modes

java.awt.HeadlessException at startup

Cause: The --headless flag is absent, or a wrapper script is stripping it before calling the binary.

Diagnostic:

grep -n "headless" your_launch_script.py

Fix: Add --headless as the first positional argument after the binary name. Verify it appears in the cmd list logged by the controller.

KeyError: 'Address' when reading the CSV

Cause: The export was read with encoding='utf-8' instead of encoding='utf-8-sig'. The BOM prefix () is prepended to the first column name, producing 'Address'.

Diagnostic:

import pandas as pd
df = pd.read_csv("internal_all.csv", encoding="utf-8", nrows=0)
print(repr(df.columns[0]))   # expect 'Address' if BOM is present

Fix: Change the read_csv call to encoding='utf-8-sig'.

Parallel crawl jobs overwrite each other's configs

Cause: Both jobs write to the same /tmp/dynamic.seospider path, so the second job's config overwrites the first mid-crawl.

Diagnostic: Check the container's /tmp directory for a single .seospider file when two crawls should be running concurrently.

Fix: Use domain-scoped config paths as shown in stage 2: os.path.join(output_dir, f"{target_domain}.seospider"). Each crawl job should write to its own output directory.

FAQ

Why does Screaming Frog silently crawl in demo mode (500 URL cap) even with a valid licence?

The CLI only reads SF_LICENSE_KEY from the process environment, not from a config file. Confirm the variable is actually present inside the container:

docker run --rm -e SF_LICENSE_KEY="${SF_LICENSE_KEY}" your-sf-image:21.4 \
  bash -c 'echo "Key length: ${#SF_LICENSE_KEY}"'

If the length is 0, the --env flag was applied to the wrong command or the variable is unset in the calling shell.

How do I prevent a second scheduled crawl from starting while one is already running?

Write the subprocess PID to a lockfile and check for it before launch. When setting up a cron job for weekly site crawls this pattern prevents overlapping crawl runs from exhausting memory:

import os, sys
from pathlib import Path

LOCKFILE = Path("/data/crawls/.crawl.lock")

if LOCKFILE.exists():
    pid = LOCKFILE.read_text().strip()
    sys.exit(f"Crawl already running (PID {pid}) — lockfile at {LOCKFILE}")

LOCKFILE.write_text(str(os.getpid()))
try:
    # ... run crawl ...
    pass
finally:
    LOCKFILE.unlink(missing_ok=True)
What is the safest way to pass the licence key inside a GitHub Actions workflow?

Store the key as an Actions secret (SF_LICENSE_KEY) and inject it via the env: block on the crawl step only — not at the job level — so it is not exposed to unrelated steps:

- name: Run Screaming Frog crawl
  env:
    SF_LICENSE_KEY: ${{ secrets.SF_LICENSE_KEY }}
  run: python3 -m crawler.controller
How do I retain only the last N crawl artifacts without a manual cleanup step?

Use an S3 Lifecycle policy to expire objects older than your retention window, or tag each upload with the crawl date and run a prefixed list-and-delete step. See storing and versioning crawl artifacts in cloud storage for a complete retention and versioning pattern including S3 object versioning and Glacier transition rules.