Integrating Audit Alerts with Incident Management #

Problem Framing #

A threshold breach that only shows up as a red cell in a dashboard is a breach nobody actions until someone happens to look. The scoring and routing logic described in Configuring Alert Thresholds and Routing decides when a regression is severe enough to matter — this guide, part of Monitoring, Alerting & Remediation, covers what happens next: turning that decision into a page, a Slack message, or a ticket that a human is actually on the hook to acknowledge. Get the wiring wrong and you get one of two failure modes: silent regressions that never reach on-call, or an incident channel so noisy that real pages get swiped away along with the noise.

The pattern below treats the incident platform as a dumb, reliable notification and escalation layer, and keeps all of the judgment — is this real, is it severe, has it recovered — in the scoring pipeline. The emitter's only job is to translate a confirmed state change into an Events API call with a payload that is deterministic, deduplicated, and resolvable.

---

Prerequisites & Environment Setup #

The emitter needs network egress to the incident platform's API, a per-environment routing key issued by that platform, and a place to persist emitted events for audit purposes. Store every credential as an environment variable injected at deploy time — never in a committed config file.

Pinned tool versions

Required environment variables

# /etc/incident-emitter/env — sourced by the emitter's systemd unit
export PAGERDUTY_ROUTING_KEY="${PAGERDUTY_ROUTING_KEY:?missing routing key}"
export PAGERDUTY_EVENTS_URL="https://events.pagerduty.com/v2/enqueue"
export SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL:?missing webhook}"
export INCIDENT_LOG_DB="/var/lib/incident-emitter/events.db"
export EMIT_ENV="production"
export TZ="UTC"

Never hardcode PAGERDUTY_ROUTING_KEY in a script or commit it to version control — treat it exactly like a database password. Rotate it through the incident platform's admin console and redeploy the environment file, not the code.

---

Step 1 — Initialize the Webhook Emitter #

The emitter is a thin translation layer. It receives a confirmed breach (or a confirmed recovery) from the scoring pipeline and turns it into a well-formed Events API v2 request. Keep the HTTP concerns — retries, timeouts, backoff — isolated from the payload-building logic so each can be tested independently.

# /opt/incident-emitter/emit.py
# Requires: requests==2.32.3, tenacity==8.3.0
from __future__ import annotations
import hashlib
import json
import os
import time
from dataclasses import dataclass, asdict
from typing import Literal

import requests
from tenacity import retry, stop_after_attempt, wait_exponential

EVENTS_URL = os.environ["PAGERDUTY_EVENTS_URL"]
ROUTING_KEY = os.environ["PAGERDUTY_ROUTING_KEY"]

Severity = Literal["critical", "warning", "info"]
Action = Literal["trigger", "resolve"]


@dataclass
class BreachEvent:
    metric: str            # e.g. "lcp_p95"
    segment: str            # e.g. "checkout"
    environment: str        # e.g. "production"
    severity: Severity
    summary: str
    source: str = "site-health-audit-scoring"


def build_dedup_key(event: BreachEvent) -> str:
    """Stable across retriggers of the same underlying condition."""
    raw = f"{event.metric}:{event.segment}:{event.environment}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]


def build_payload(event: BreachEvent, action: Action) -> dict:
    return {
        "routing_key": ROUTING_KEY,
        "event_action": action,
        "dedup_key": build_dedup_key(event),
        "payload": {
            "summary": event.summary,
            "severity": event.severity,
            "source": event.source,
            "component": event.segment,
            "custom_details": {
                "metric": event.metric,
                "environment": event.environment,
            },
        },
    }


@retry(stop=stop_after_attempt(4), wait=wait_exponential(multiplier=1, min=1, max=15))
def post_event(payload: dict) -> requests.Response:
    resp = requests.post(EVENTS_URL, json=payload, timeout=8)
    resp.raise_for_status()
    return resp


def emit_breach(event: BreachEvent) -> str:
    payload = build_payload(event, "trigger")
    resp = post_event(payload)
    return payload["dedup_key"]


def emit_resolve(event: BreachEvent) -> str:
    payload = build_payload(event, "resolve")
    resp = post_event(payload)
    return payload["dedup_key"]

Note that build_dedup_key never touches the timestamp or the raw metric value — only the identity of the condition. Two calls to emit_breach for the same (metric, segment, environment) tuple, minutes or hours apart, produce the same dedup_key, so the incident platform correctly merges retriggers into a single open incident rather than spawning duplicates.

---

Step 2 — Core Parameters: Severity, Dedup, and Auto-Resolve #

Four fields govern how the incident platform groups, prioritizes, and closes an alert. Get these wrong and either incidents fragment (weak dedup) or real regressions get buried under stale open pages (missing auto-resolve).

Key parameters table

The Events API v2 payload produced by build_payload for a confirmed critical breach looks like this on the wire:

{
  "routing_key": "R0ABC123EXAMPLEROUTINGKEY",
  "event_action": "trigger",
  "dedup_key": "9f2a7c1e4b6d8035a1c9f7e2b3d4a6c8",
  "payload": {
    "summary": "LCP p95 breached critical threshold on checkout segment",
    "severity": "critical",
    "source": "site-health-audit-scoring",
    "component": "checkout",
    "custom_details": {
      "metric": "lcp_p95",
      "environment": "production"
    }
  }
}

The auto_resolve behavior comes from calling emit_resolve with a BreachEvent carrying the same metric, segment, and environment as the original trigger — the identical inputs to build_dedup_key guarantee the resolve event targets the correct open incident.

---

Alert-to-Incident Flow #

The diagram below traces a single alert from the moment the scoring engine confirms a breach through escalation and, eventually, automatic resolution once a later audit run confirms recovery.

---

Step 3 — Execution & Escalation #

Fire the emitter from the same run that confirms the breach, not from an ad hoc script — that keeps the confirmation logic (consecutive-run counting, cooldown) in one place. Escalation policy configuration lives on the incident platform side, but the on-call schedule's timezone has to be explicit, because a schedule stored in local time silently shifts every daylight-saving transition.

# /opt/incident-emitter/run_on_breach.py
# Requires: requests==2.32.3
from __future__ import annotations
import os
import sqlite3
import time
from datetime import datetime, timezone

from emit import BreachEvent, emit_breach, emit_resolve

CONFIRM_RUNS = int(os.environ.get("CONFIRM_RUNS", "2"))
DB_PATH = os.environ["INCIDENT_LOG_DB"]


def confirmed_breach_streak(metric: str, segment: str) -> int:
    """Count consecutive breach observations for this condition from the score history table."""
    con = sqlite3.connect(DB_PATH)
    row = con.execute(
        "SELECT streak FROM breach_streaks WHERE metric = ? AND segment = ?",
        (metric, segment),
    ).fetchone()
    con.close()
    return row[0] if row else 0


def handle_scoring_result(metric: str, segment: str, environment: str, breached: bool, severity: str) -> None:
    event = BreachEvent(metric=metric, segment=segment, environment=environment, severity=severity,
                         summary=f"{metric} breached {severity} threshold on {segment}")
    streak = confirmed_breach_streak(metric, segment)

    if breached and streak >= CONFIRM_RUNS:
        dedup_key = emit_breach(event)
        print(f"[{datetime.now(timezone.utc).isoformat()}] TRIGGER dedup_key={dedup_key}")
    elif not breached and streak == 0:
        dedup_key = emit_resolve(event)
        print(f"[{datetime.now(timezone.utc).isoformat()}] RESOLVE dedup_key={dedup_key}")

Escalation policy on the platform side should page the primary on-call engineer immediately for critical severity, wait a fixed acknowledgment window (5 minutes is a reasonable default for production-facing regressions), then escalate to the secondary. Configure the on-call rotation's schedule with an explicit TZ field rather than relying on the platform's account-level default timezone — a rotation defined in America/New_York behaves correctly across the DST boundary; one defined as a fixed UTC offset silently pages the wrong person for half the year.

For the concrete PagerDuty wiring — routing key setup, escalation policy IDs, and a worked trigger/resolve pair — see Routing Audit Alerts to PagerDuty. Once an incident is acknowledged, the responder typically hands off to a remediation procedure; Writing Remediation Playbooks for Audit Failures covers turning that acknowledgment into a runnable fix rather than a manual scramble.

---

Step 4 — Artifact Capture & Incident Log Retention #

Every emitted event — trigger or resolve, successful or failed — should land in an append-only log before the pipeline considers the run complete. This log is the ground truth for reconciling "did we actually get paged" questions during a postmortem, and it is what you diff against score history when tracking metric trends across release cycles to confirm an incident correlates with a real regression rather than a noisy sensor.

# /opt/incident-emitter/log_event.py
# Requires: sqlite3 (stdlib)
from __future__ import annotations
import sqlite3
from datetime import datetime, timezone

SCHEMA = """
CREATE TABLE IF NOT EXISTS incident_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    emitted_at TEXT NOT NULL,
    dedup_key TEXT NOT NULL,
    metric TEXT NOT NULL,
    segment TEXT NOT NULL,
    action TEXT NOT NULL,
    severity TEXT NOT NULL,
    http_status INTEGER,
    response_body TEXT
);
"""


def log_event(db_path: str, dedup_key: str, metric: str, segment: str,
              action: str, severity: str, http_status: int, response_body: str) -> None:
    con = sqlite3.connect(db_path)
    con.execute(SCHEMA)
    con.execute(
        "INSERT INTO incident_log (emitted_at, dedup_key, metric, segment, action, severity, http_status, response_body) "
        "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
        (datetime.now(timezone.utc).isoformat(), dedup_key, metric, segment, action, severity, http_status, response_body),
    )
    con.commit()
    con.close()

Retain the incident log for at least 180 days — longer than the score-artifact retention window — so a delayed postmortem can still reconstruct exactly which incidents fired around a given release. Back the SQLite file up to object storage nightly rather than trusting local disk on the emitter host.

---

Verification Checklist #

Run through this list after wiring a new severity tier or changing routing configuration, and again after any incident-platform credential rotation:

1. Send a synthetic trigger with a test dedup_key and confirm it appears as an open incident in the platform UI within 30 seconds.
2. Send a matching resolve with the same dedup_key and confirm the same incident closes rather than a new one opening.
3. Verify escalation_policy_id on the routing config matches an active policy — a stale or deleted policy ID causes the platform to accept the event but never page anyone.
4. Confirm the on-call schedule's timezone field is set explicitly, not inherited from account defaults: curl -s -H "Authorization: Token token=$PD_API_TOKEN" https://api.pagerduty.com/schedules/$SCHEDULE_ID | jq '.schedule.time_zone'.
5. Query the incident log for the last 24 hours and confirm every trigger action has either a matching resolve or is still an open, actively-breaching condition: sqlite3 "$INCIDENT_LOG_DB" "SELECT dedup_key, action, count(*) FROM incident_log WHERE emitted_at > datetime('now','-1 day') GROUP BY dedup_key;".
6. Confirm CONFIRM_RUNS is at least 2 in production — a value of 1 means a single noisy sample can page on-call at 3am.

---

Troubleshooting #

Incidents open but nobody gets paged

Root cause: the routing_key points at a valid integration, but the escalation policy attached to that service has no active on-call schedule, or every layer of the schedule is empty.

curl -s -H "Authorization: Token token=$PD_API_TOKEN" \
  "https://api.pagerduty.com/escalation_policies/$POLICY_ID" | jq '.escalation_policy.escalation_rules'

Fix: assign an active schedule to every escalation rule layer and re-test with a synthetic trigger before trusting the pipeline again.

---

Duplicate incidents for the same regression

Root cause: dedup_key includes a component that changes between retriggers — most commonly a timestamp or a raw metric value accidentally concatenated into the hash input.

python3 -c "
from emit import BreachEvent, build_dedup_key
e1 = BreachEvent(metric='lcp_p95', segment='checkout', environment='production', severity='critical', summary='x')
e2 = BreachEvent(metric='lcp_p95', segment='checkout', environment='production', severity='critical', summary='y')
print(build_dedup_key(e1) == build_dedup_key(e2))
"

Fix: rebuild dedup_key from only identity fields (metric, segment, environment) and redeploy; merge any duplicate open incidents manually in the platform UI.

---

Auto-resolve never fires even after the metric recovers

Root cause: the resolve path requires the exact same dedup_key as the original trigger, but a code change altered the segment-naming convention between the two calls (e.g. checkout vs /checkout).

sqlite3 "$INCIDENT_LOG_DB" \
  "SELECT dedup_key, action, segment FROM incident_log WHERE metric='lcp_p95' ORDER BY emitted_at DESC LIMIT 5;"

Fix: normalize segment identifiers (trim slashes, lowercase) in one shared helper used by both the trigger and resolve paths.

---

Alert storm during a deploy window

Root cause: expected metric wobble during a release crosses the confirmation threshold and pages on-call for a regression that resolves itself within minutes.

grep "TRIGGER" /var/log/incident-emitter/*.log | tail -20

Fix: gate emission behind a deploy-window suppression before it reaches the emitter — see the threshold-and-routing configuration in Configuring Alert Thresholds and Routing rather than trying to fix it at the incident-platform layer.

---

Events API calls fail with HTTP 429

Root cause: the incident platform rate-limits Events API calls per routing key, and a burst of many segments breaching simultaneously exceeds it.

python3 -c "
from log_event import log_event
import sqlite3
con = sqlite3.connect('$INCIDENT_LOG_DB')
print(con.execute(\"SELECT count(*) FROM incident_log WHERE http_status = 429 AND emitted_at > datetime('now','-1 hour')\").fetchone())
"

Fix: batch same-severity breaches from a single scoring run into one incident with a segment list in custom_details rather than one call per segment, or add jittered client-side rate limiting ahead of the retry wrapper.

---

Emitter retries succeed but the incident log shows a gap

Root cause: tenacity's retry wrapper eventually succeeds on attempt 3 or 4, but log_event was only called once, before the retries ran, logging a stale failure status.

sqlite3 "$INCIDENT_LOG_DB" "SELECT * FROM incident_log WHERE http_status != 200 ORDER BY emitted_at DESC LIMIT 10;"

Fix: move the log_event call to after post_event returns (or exhausts retries), logging the final http_status, not the first attempt's.

---

How do I stop a flapping metric from opening and closing the same incident over and over?

Require a breach to hold across N consecutive confirmed runs before the emitter fires, and apply a cooldown after resolve so a metric that recovers for one run and regresses on the next does not immediately reopen. Combine this with a stable dedup_key so PagerDuty or Opsgenie deduplicates retriggers into the same incident rather than opening duplicates.

What should the dedup_key be built from?

Hash the metric name, the URL segment or template it applies to, and the environment, but never the timestamp or the raw metric value. A dedup_key of sha256(metric+segment+environment) stays stable across every retrigger of the same underlying problem, so the platform correctly groups repeat pages into one incident instead of fragmenting them.

Does auto-resolve fire on any single passing sample, or does it require a confirmed healthy run?

Auto-resolve should require a confirmed healthy run, not a single sample, for the same reason breaches require confirmation: one clean reading during a noisy period is not proof of recovery. Gate resolve events behind the same run-window logic that gates triggers so incidents do not flap closed and immediately reopen.

Can the same audit alert route to both PagerDuty and Slack at once?

Yes. Treat each destination as an independent emit call driven by the same confirmed-breach event, each with its own dedup_key namespaced by platform. A PagerDuty page for the on-call engineer and a parallel Slack post for team visibility do not need to share dedup state, since they serve different audiences with different acknowledgment semantics.

---

Related #

• Monitoring, Alerting & Remediation — parent section covering the detect-to-remediate loop
• Routing Audit Alerts to PagerDuty — worked Events API v2 trigger/resolve example with dedup_key
• Configuring Alert Thresholds and Routing — how severity tiers and routing decisions are produced upstream of this emitter
• Writing Remediation Playbooks for Audit Failures — what an acknowledged incident hands off to