Writing Remediation Playbooks for Audit Failures #

Problem Framing #

An audit that only detects problems is half a system. Every recurring finding — a redirect chain that grew another hop, a sitemap entry pointing at a page that now 404s, a crawl budget quietly drained by faceted-navigation URLs — eventually gets fixed by someone typing commands from memory into a terminal at 11 p.m. The fix works, but it isn't written down, it isn't versioned, and the next time the same finding fires, a different engineer reinvents it slightly differently. Six months later nobody can say with confidence what "the fix" for a given finding actually does, and two competing tribal-knowledge fixes start contradicting each other in the same codebase.

A remediation playbook closes that gap. It is a small, parameterized, idempotent program that takes a finding type and a target scope, proves the finding is real, applies the minimal fix, and proves the fix worked — all without a human needing to remember the procedure. This guide covers the full anatomy of that program, sitting underneath Monitoring, Alerting & Remediation as the layer that turns a routed alert into a completed, auditable action rather than a page that a human has to manually resolve.

Three concrete finding types anchor the examples below, and each has its own dedicated companion guide once you need the full implementation: fixing broken redirect chains, resolving orphaned sitemap entries, and recovering from crawl budget exhaustion. This page focuses on the shared skeleton all three (and any future playbook) should be built on.

---

Prerequisites & Environment Setup #

A playbook runner needs a stable execution environment, a place to read its own config from version control rather than from local edits, and credentials scoped narrowly enough that a bug in one playbook cannot cascade into an unrelated system.

Pinned tool versions

Required environment variables

# /etc/remediation/env — sourced by the playbook runner
export PLAYBOOK_REPO="/opt/remediation/playbooks"
export AUDIT_TRAIL_BUCKET="gs://site-audit-remediation-prod"
export CMS_API_TOKEN="${CMS_API_TOKEN:?set in secrets manager}"
export ALERT_SOURCE_URL="https://alerts.internal/api/v1/findings"
export REMEDIATION_ENV="production"
export TZ="UTC"

Store playbook definitions in the same repository that holds the alert-routing config described in configuring alert thresholds and routing, so a single pull request can adjust both a threshold and the playbook it triggers, reviewed together.

---

Step 1 — Playbook Skeleton #

Every playbook is two files: a YAML definition describing its stages and guardrails, and a thin bash runner that the alert router invokes with a finding type and a target. The runner never contains business logic — it only sequences pre-check, dry-run, fix, and post-verify, and it always respects the dry_run flag.

# /opt/remediation/playbooks/broken_redirect_chain.yaml
playbook_id: broken_redirect_chain_v3
finding_type: broken_redirect_chain
description: Collapse a multi-hop redirect chain to a single 301.
pre_checks:
  - name: chain_still_present
    command: "scripts/check_redirect_chain.sh {{ target_url }}"
  - name: no_active_deploy
    command: "scripts/deploy_lock_status.sh"
fix:
  command: "scripts/collapse_redirect.sh {{ target_url }} {{ final_url }}"
post_verify:
  - name: single_hop_confirmed
    command: "scripts/check_redirect_chain.sh {{ target_url }} --max-hops 1"
rollback:
  command: "scripts/restore_redirect_snapshot.sh {{ target_url }} {{ snapshot_id }}"

#!/usr/bin/env bash
# /opt/remediation/run_playbook.sh
set -euo pipefail

FINDING_TYPE="${1:?usage: run_playbook.sh <finding_type> <target> [--live]}"
TARGET="${2:?target required}"
LIVE=false
[[ "${3:-}" == "--live" ]] && LIVE=true

DEFINITION="${PLAYBOOK_REPO}/playbooks/${FINDING_TYPE}.yaml"
[[ -f "${DEFINITION}" ]] || { echo "No playbook for ${FINDING_TYPE}" >&2; exit 2; }

echo "== Playbook: ${FINDING_TYPE} | target: ${TARGET} | live: ${LIVE} =="

# 1. Pre-checks — every one must pass before the fix stage runs
yq '.pre_checks[].command' "${DEFINITION}" | while read -r check; do
  eval "${check/\{\{ target_url \}\}/${TARGET}}" || {
    echo "Pre-check failed, aborting: ${check}" >&2
    exit 3
  }
done

# 2. Dry-run diff — always computed, even in live mode, for the audit trail
scripts/diff_target_state.sh "${TARGET}" --dry-run > "/tmp/${FINDING_TYPE}_diff.txt"
cat "/tmp/${FINDING_TYPE}_diff.txt"

if [[ "${LIVE}" == false ]]; then
  echo "Dry run complete — no changes applied. Re-run with --live to execute."
  exit 0
fi

# 3. Fix — only reached in live mode, after a clean dry-run diff
FIX_CMD=$(yq '.fix.command' "${DEFINITION}")
eval "${FIX_CMD/\{\{ target_url \}\}/${TARGET}}"

# 4. Post-verify — the fix is not considered successful until this passes
yq '.post_verify[].command' "${DEFINITION}" | while read -r check; do
  eval "${check/\{\{ target_url \}\}/${TARGET}}" || {
    echo "Post-verify failed — invoking rollback" >&2
    ROLLBACK_CMD=$(yq '.rollback.command' "${DEFINITION}")
    eval "${ROLLBACK_CMD/\{\{ target_url \}\}/${TARGET}}"
    exit 4
  }
done

echo "== Playbook completed and verified: ${FINDING_TYPE} on ${TARGET} =="

The --live flag is the only thing separating a safe preview from an actual mutation. Never let a playbook accept dry_run: false from an untrusted or automated caller without also checking the alert that triggered it is still active — a stale alert triggering a live fix against a target that has since resolved itself is a common source of unnecessary churn.

---

Step 2 — Core Configuration Parameters #

Playbook behavior should be entirely visible in configuration, not buried in script logic. The parameters below apply to every playbook regardless of finding type, and the alert router reads them before deciding whether to invoke run_playbook.sh at all.

# /opt/remediation/config/runner_defaults.yaml
defaults:
  dry_run: true
  rollback_on_failure: true
  max_targets_per_run: 50
  cooldown_minutes: 20

overrides:
  broken_redirect_chain:
    max_targets_per_run: 25   # redirect rewrites touch shared routing config; keep batches small
  crawl_budget_exhaustion:
    max_targets_per_run: 200  # robots/exclude rule edits are lower risk per target

Capping max_targets_per_run matters more than it looks. A playbook that scopes correctly against ten known-bad URLs but is accidentally invoked with a target_scope glob matching ten thousand URLs should fail loudly at the cap, not silently rewrite the entire site in one pass.

---

Playbook Execution Flow #

The diagram below shows the fixed sequence every playbook follows regardless of finding type, and the two terminal states — close or roll back — that post-verify decides between.

---

Step 3 — Execution & Scheduling #

Playbooks should never run on a fixed schedule; they run in response to a confirmed alert, after the alert has survived its cooldown window so a single transient scrape does not trigger a live fix. Wire the trigger to the same routing layer covered in configuring alert thresholds and routing — the router calls the playbook runner with the finding type and target scope once severity and cooldown criteria are both satisfied.

#!/usr/bin/env bash
# /opt/remediation/trigger_from_alert.sh
set -euo pipefail

ALERT_JSON="${1:?path to confirmed alert JSON payload}"
FINDING_TYPE=$(jq -r '.finding_type' "${ALERT_JSON}")
TARGET=$(jq -r '.target_url' "${ALERT_JSON}")
CONFIRMED_MINUTES=$(jq -r '.minutes_since_first_seen' "${ALERT_JSON}")
COOLDOWN=20

if (( CONFIRMED_MINUTES < COOLDOWN )); then
  echo "Alert younger than cooldown (${CONFIRMED_MINUTES}m < ${COOLDOWN}m) — skipping." >&2
  exit 0
fi

LOCKFILE="/var/lock/remediation-${FINDING_TYPE}.lock"
exec 200>"${LOCKFILE}"
flock -n 200 || { echo "Playbook ${FINDING_TYPE} already running — exiting." >&2; exit 1; }

/opt/remediation/run_playbook.sh "${FINDING_TYPE}" "${TARGET}" --live

Note the lock is scoped per finding type, not globally — a redirect-chain playbook running against one section should not block an unrelated sitemap playbook from starting against a different target. Scope locks as narrowly as the blast radius of the fix itself.

For environments where the trigger runs inside CI rather than a long-lived host, express the same flock guard as a pipeline-level concurrency key, exactly as described for scoring jobs, so two pipeline runs triggered by overlapping alerts cannot race on the same lock resource.

---

Step 4 — Artifact Capture & Audit Trail #

A playbook run that leaves no record is barely better than a manual fix. Capture the target's state immediately before the fix, the state immediately after, and the diff between them, then write all three to an append-only audit trail tagged with the playbook version and the Git SHA that produced it.

# /opt/remediation/audit_trail.py
# Requires: none beyond the standard library
from __future__ import annotations
import hashlib
import json
import subprocess
from datetime import datetime, timezone
from pathlib import Path


def git_sha(repo_path: str) -> str:
    return subprocess.check_output(
        ["git", "-C", repo_path, "rev-parse", "--short", "HEAD"]
    ).decode().strip()


def write_audit_record(
    finding_type: str,
    target: str,
    before_state: dict,
    after_state: dict,
    playbook_repo: str,
    bucket: str,
) -> str:
    """Write a single before/after audit record for one playbook run."""
    run_time = datetime.now(tz=timezone.utc)
    record = {
        "finding_type": finding_type,
        "target": target,
        "playbook_sha": git_sha(playbook_repo),
        "run_timestamp": run_time.isoformat(),
        "before_state": before_state,
        "after_state": after_state,
        "before_checksum": hashlib.sha256(
            json.dumps(before_state, sort_keys=True).encode()
        ).hexdigest(),
        "after_checksum": hashlib.sha256(
            json.dumps(after_state, sort_keys=True).encode()
        ).hexdigest(),
    }
    partition = run_time.strftime("%Y/%m/%d")
    out_path = Path(f"{bucket}/audit-trail/{partition}/{finding_type}_{run_time.timestamp():.0f}.json")
    out_path.parent.mkdir(parents=True, exist_ok=True)
    out_path.write_text(json.dumps(record, indent=2))
    return str(out_path)

Retention — keep audit-trail records for 180 days minimum, longer than raw crawl artifacts, since these records are what a post-incident review or a compliance auditor asks for months after the underlying crawl data has already aged out of the storage tier described in storing and versioning crawl artifacts in cloud storage.

---

Verification Checklist #

Run through this list after every live playbook invocation, before treating a finding as closed:

1. Confirm the runner log contains the line == Playbook completed and verified: — its absence means the run exited before post-verify ran.
2. Re-fetch the target and confirm the specific condition the finding described is gone (e.g. curl -sIL shows one redirect hop, not a chain).
3. Confirm an audit-trail JSON record exists for this run at the expected date partition and that before_checksum differs from after_checksum — an identical checksum means the fix was a no-op.
4. Diff the target's health score before and after the fix; a live-mode run that shows no score improvement warrants investigation before the finding is closed.
5. Confirm the alert that triggered the run has transitioned to resolved in the alert router, not just marked done locally in the playbook log.
6. Spot-check that max_targets_per_run was not hit — if the run stopped at the cap, schedule a follow-up run rather than assuming the finding is fully resolved.

---

Troubleshooting #

Playbook exits at the pre-check stage on every invocation

Root cause: the pre-check command references a target format the alert payload no longer sends — commonly a scheme or trailing-slash mismatch after a router config change.

# Run the pre-check in isolation against a known target
scripts/check_redirect_chain.sh "https://example.com/old-path/" ; echo "exit: $?"

Fix: normalize the target URL the same way in both the alert payload and the pre-check script — strip trailing slashes and lowercase the scheme/host before comparison.

---

Dry-run diff is empty even though the finding is confirmed active

Root cause: diff_target_state.sh is reading cached state from a previous run instead of fetching live state, usually because a local cache TTL was set too generously.

# Force a live fetch, bypassing any local cache
scripts/diff_target_state.sh "${TARGET}" --dry-run --no-cache

Fix: set the cache TTL for pre-fix state reads to zero, or route the read through a Cache-Control: no-cache request so the dry-run always reflects current reality.

---

Live run applies the fix but post-verify still fails

Root cause: post-verify re-crawled the target before the fix had propagated — CDN edge caches or a load balancer with sticky config reloads can lag the origin by tens of seconds.

# Insert a bounded wait with retry before post-verify
for i in $(seq 1 6); do
  scripts/check_redirect_chain.sh "${TARGET}" --max-hops 1 && break
  sleep 10
done

Fix: add a short bounded retry loop between fix and post-verify rather than a single immediate check, and cap it — an unbounded retry masks a genuinely failed fix.

---

Two playbook runs collide on the same target and produce conflicting states

Root cause: the flock guard is scoped to the finding type but two different finding types both touch overlapping URL scope (for example a redirect fix and a sitemap fix on the same path).

# List active locks across all finding types
ls -la /var/lock/remediation-*.lock

Fix: add a shared target-level lock keyed on a normalized URL prefix, in addition to the per-finding-type lock, so overlapping scopes serialize correctly.

---

Rollback runs but the finding reappears within minutes

Root cause: rollback restored the pre-fix state but the underlying cause (a CMS template or CDN rule generating the bad redirect) was never addressed, so the same finding regenerates on the next crawl.

# Confirm whether the finding is generated by a template rather than a one-off page
grep -rl "old-path" /opt/cms/templates/

Fix: treat a repeat finding on the same target within 24 hours of a rollback as a signal to escalate past the playbook to a manual template-level fix, rather than re-running the same playbook again.

---

Audit-trail write fails silently and the run reports success anyway

Root cause: the runner does not check the exit code of the audit-trail write step, so a bucket permissions error or a full disk gets swallowed.

# Confirm the audit-trail write actually landed
gsutil ls "gs://site-audit-remediation-prod/audit-trail/$(date -u +%Y/%m/%d)/" | tail -5

Fix: make the audit-trail write a hard requirement for a successful exit code — a playbook that cannot record what it did should be treated the same as a playbook that failed.

---

Common Mistakes #

• Skipping the dry-run diff in a hurry to close an incident faster — the diff is often what catches a target_scope glob that matched far more URLs than intended.
• Writing fix logic that is not idempotent, so a retried run after a network blip re-applies a change on top of itself and produces a state nobody predicted.
• Letting max_targets_per_run default to unlimited "just for this one incident" — that exception is exactly the run that goes wrong.
• Treating rollback as an afterthought instead of a first-class stage with its own pre-checks; a rollback command that itself fails midway is worse than the original finding.
• Forgetting to tag audit-trail records with the playbook's own version, which makes it impossible to tell which fix logic ran when a playbook is later revised.

---

Should a remediation playbook default to dry-run mode?

Yes. Every playbook should default dry_run to true so a misconfigured target_scope or an untested playbook version cannot silently mutate production. Require an explicit --live flag or dry_run: false in the invocation config before any write occurs, and log the flag's value at the top of every run so the log itself proves whether a given execution was safe to ignore or worth reviewing.

How do I make a remediation playbook idempotent?

Check current state before acting and skip targets already in the desired state. A redirect-collapse playbook, for example, should read the existing rule first and only rewrite it if the destination differs from the computed final URL. Idempotent playbooks are safe to re-run after a partial failure without compounding the fix, which matters most during the exact moment — a flaky retry, an on-call engineer re-triggering by hand — when you can least afford a second, unexpected mutation.

What belongs in the pre-check stage versus the fix stage?

Pre-checks validate assumptions and acquire locks: confirm the target still exhibits the finding, confirm no conflicting deploy is in flight, and confirm a rollback snapshot can be taken. The fix stage should contain only the minimal mutation itself, with no validation logic, so it stays easy to audit line by line and easy to reason about when something goes wrong mid-execution.

How long should before/after audit trail artifacts be retained?

Retain playbook audit-trail artifacts for at least 180 days, longer than typical crawl-artifact retention, because remediation history is frequently needed for post-incident reviews and compliance audits well after the underlying crawl data has expired. If storage cost is a concern, tier audit-trail JSON to cold storage after 30 days rather than deleting it outright.

---

Related #

• Monitoring, Alerting & Remediation — parent section covering the full detect-to-remediate loop
• Fixing Broken Redirect Chains — full implementation of the redirect-collapse playbook
• Resolving Orphaned Sitemap Entries — reconciling sitemap URLs against the live crawl graph
• Recovering from Crawl Budget Exhaustion — reclaiming budget burned on low-value URL patterns