Configuring Alert Thresholds and Routing #

Problem Framing #

A scoring pipeline that produces a composite health score every hour is only useful if a regression in that score reaches a human before it reaches a customer. Most teams get this wrong in one of two directions. Either every score dip below some arbitrary number pages the on-call engineer at 3 a.m. for a blip that self-corrects on the next crawl, or thresholds are set so loose that a genuine regression sits unnoticed for days because nobody wants to be the person who tuned the alert that cried wolf. Both failure modes have the same root cause: thresholds and routing were bolted onto the scoring pipeline as an afterthought instead of being designed as a first-class configuration with the same rigor as the scoring algorithm itself.

This guide is part of Monitoring, Alerting & Remediation and covers the threshold-and-routing layer specifically — the logic that sits between a computed health score and a notification landing in a channel. It starts with static severity tiers you can ship in an afternoon, adds the configuration parameters that make routing predictable (dedup windows, cooldowns, channel mapping), and ends with the artifact trail an incident review will need six months from now when someone asks "why didn't we get paged for this." The goal is a threshold configuration that is versioned, testable, and boring in the best sense — every alert that fires is one you can explain by pointing at a line in a config file.

---

Prerequisites & Environment Setup #

The threshold-and-routing layer is intentionally decoupled from the scoring pipeline itself — it reads scored output and reacts to it, but does not recompute anything. That separation keeps threshold changes reviewable independently of scoring-algorithm changes, and it means the routing service can be redeployed without triggering a full rescore.

Pinned tool versions

Required environment variables

# /etc/alert-routing/env  — sourced by the routing service on startup
export SCORING_ARTIFACT_BUCKET="gs://site-audit-artifacts-prod"
export THRESHOLD_CONFIG="/opt/alert-routing/thresholds.json"
export ROUTING_CONFIG="/opt/alert-routing/routing.yaml"
export ALERT_STATE_REDIS_URL="redis://alert-state.internal:6379/2"
export ALERT_HISTORY_LOG_DIR="/var/log/site-health-alerts"
export PAGERDUTY_ROUTING_KEY="${PAGERDUTY_ROUTING_KEY:?unset}"
export SLACK_WEBHOOK_SEO="${SLACK_WEBHOOK_SEO:?unset}"
export TZ="UTC"

Commit thresholds.json and routing.yaml to version control alongside the scoring pipeline's own config, not to a separate operations repository. When a threshold change and a scoring change ship in the same pull request, reviewers can see the full picture — a weight-matrix edit that shifts the composite score distribution should almost always come with a corresponding threshold review, and keeping them in one diff makes that connection visible instead of implicit.

Validate the threshold file against a JSON Schema in CI before it can merge:

python3 -m jsonschema \
  --instance /opt/alert-routing/thresholds.json \
  /opt/alert-routing/thresholds.schema.json

---

Step 1 — Initialize the Threshold Configuration #

Begin with static severity tiers. Each tier defines a score floor — the composite health score below which a finding at that severity is considered active — and the channels it should reach. Keep the file flat and readable; this is the document an incident commander opens first during a postmortem.

// /opt/alert-routing/thresholds.json  (version-controlled)
{
  "version": "1.4.0",
  "tiers": {
    "critical": {
      "score_floor": 0.55,
      "channels": ["pagerduty-sre", "slack-incidents"],
      "dedup_window_minutes": 15,
      "cooldown_minutes": 30
    },
    "warning": {
      "score_floor": 0.75,
      "channels": ["slack-seo-team"],
      "dedup_window_minutes": 30,
      "cooldown_minutes": 120
    },
    "info": {
      "score_floor": 0.90,
      "channels": ["slack-monitoring-digest"],
      "dedup_window_minutes": 60,
      "cooldown_minutes": 1440
    }
  },
  "section_overrides": {
    "checkout": {
      "critical": { "score_floor": 0.70 },
      "warning": { "score_floor": 0.85 }
    },
    "blog": {
      "critical": { "score_floor": 0.45 }
    }
  }
}

section_overrides exists because a single global floor treats a checkout flow the same as a low-traffic blog archive, which is rarely correct — see Calibrating Error Thresholds for Different Site Sections for the reasoning behind per-section tuning. The override block is merged on top of the tier default at evaluation time, so a section that has no override simply inherits the global floor.

Evaluate tiers from most severe to least severe and stop at the first match — a score of 0.60 on the checkout section matches critical (floor 0.70 after override) before it is ever checked against warning, so a single finding produces exactly one severity classification, never two.

---

Step 2 — Core Configuration Parameters #

Three parameters do almost all of the work in this layer: which tier a score falls into, how long a repeat finding is treated as "the same alert" for deduplication purposes, and how long to wait before notifying again for that same alert. Get these three right and the routing logic underneath is mostly plumbing.

Key parameters table

Move these into a production routing file that references the threshold tiers by name rather than duplicating the floors — routing.yaml owns where alerts go and how often, while thresholds.json owns what counts as a breach. Keeping the two concerns in separate files means a channel migration (swapping a Slack webhook, say) never touches the severity logic.

# /opt/alert-routing/routing.yaml  (version-controlled)
routing:
  critical:
    channels:
      - type: pagerduty
        target: pagerduty-sre
        escalation_after_minutes: 60
      - type: slack
        target: slack-incidents
    dedup_window_minutes: 15
    cooldown_minutes: 30
  warning:
    channels:
      - type: slack
        target: slack-seo-team
    dedup_window_minutes: 30
    cooldown_minutes: 120
  info:
    channels:
      - type: slack
        target: slack-monitoring-digest
        batch: true
        digest_cron: "0 9 * * *"
    dedup_window_minutes: 60
    cooldown_minutes: 1440

defaults:
  timezone: "UTC"
  max_channels_per_finding: 3

The digest_cron field on the info channel is worth calling out: informational findings are batched into a single daily digest rather than sent as individual messages, which is why its cooldown is a full day. Sending an individual Slack message for every sub-critical drift would train the team to ignore the channel within a week.

---

Step 3 — Execution & Routing #

Evaluate the threshold config against every scoring run's output — not on a separate schedule — so alert latency is bounded by scoring latency, never by a second, independent poll interval. The evaluator reads the latest scored artifact, classifies each finding into a tier, computes a dedup key, checks the cooldown state in Redis, and dispatches.

# /opt/alert-routing/evaluate.py
# Requires: redis-py==5.0.4, PyYAML==6.0.1
from __future__ import annotations
import hashlib
import json
import os
import time
from dataclasses import dataclass

import redis
import yaml


@dataclass
class Finding:
    metric: str
    section: str
    score: float
    tier: str


def load_config() -> tuple[dict, dict]:
    with open(os.environ["THRESHOLD_CONFIG"]) as fh:
        thresholds = json.load(fh)
    with open(os.environ["ROUTING_CONFIG"]) as fh:
        routing = yaml.safe_load(fh)
    return thresholds, routing


def classify_tier(score: float, section: str, thresholds: dict) -> str | None:
    """Evaluate most-severe-first; return the first tier whose floor is breached."""
    for tier_name in ("critical", "warning", "info"):
        tier = dict(thresholds["tiers"][tier_name])
        override = thresholds.get("section_overrides", {}).get(section, {}).get(tier_name)
        if override:
            tier.update(override)
        if score < tier["score_floor"]:
            return tier_name
    return None


def dedup_key(metric: str, section: str, tier: str) -> str:
    """Stable identity for 'the same alert' — deliberately excludes the raw score."""
    raw = f"{metric}:{section}:{tier}"
    return hashlib.sha256(raw.encode()).hexdigest()[:16]


def should_notify(r: redis.Redis, key: str, cooldown_minutes: int) -> bool:
    """Atomic check-and-set: only the first caller within the cooldown wins."""
    ttl_seconds = cooldown_minutes * 60
    return bool(r.set(f"cooldown:{key}", int(time.time()), nx=True, ex=ttl_seconds))


def evaluate_and_route(scored_findings: list[Finding], thresholds: dict, routing: dict, r: redis.Redis) -> list[dict]:
    dispatched = []
    for finding in scored_findings:
        tier = classify_tier(finding.score, finding.section, thresholds)
        if tier is None:
            continue
        key = dedup_key(finding.metric, finding.section, tier)
        cooldown = routing["routing"][tier]["cooldown_minutes"]
        fired = should_notify(r, key, cooldown)
        dispatched.append({
            "dedup_key": key,
            "tier": tier,
            "metric": finding.metric,
            "section": finding.section,
            "score": finding.score,
            "fired": fired,
        })
    return dispatched

Two details matter more than the rest of the code. First, should_notify uses SET NX EX as a single atomic Redis operation rather than a separate GET followed by a SET — a check-then-act pattern across two calls is a race condition the moment two scoring workers evaluate the same finding within milliseconds of each other. Second, all cooldown TTLs and timestamps are computed in UTC (TZ=UTC from the environment), because a cooldown computed in a local timezone silently shifts by an hour twice a year at daylight-saving transitions, which is a genuinely confusing bug to chase during an on-call handoff across regions.

Once a finding's score is a percentile rank rather than a raw composite value, the same classify_tier function works unchanged against a rolling baseline instead of a fixed floor — see Setting Dynamic Alert Baselines with Rolling Percentiles for the migration path from static tiers to adaptive ones. Neither the dedup key nor the cooldown logic needs to change; only the input to classify_tier does.

For active deploys, layer a suppression check ahead of should_notify rather than inside it, so the cooldown state itself stays clean — Suppressing Alert Noise During Deploy Windows covers the deploy-tag lookup and TTL pattern in detail. Once a finding clears routing, hand it to the incident layer described in Integrating Audit Alerts with Incident Management for the actual PagerDuty or Opsgenie dispatch — this page owns classification and cooldown, not the wire format of the outbound event.

---

Threshold & Routing Architecture #

The diagram below traces one scored finding from the moment it leaves the scoring pipeline through tier classification, the dedup and cooldown gate, and out to a channel — with the suppressed path shown separately, since a suppressed alert still needs to be logged even though nothing is sent.

Both outcomes — dispatched and suppressed — write to the same history log with a fired: true|false field, which is what makes the audit trail in Step 4 trustworthy: you can reconstruct not just what fired, but what the system correctly decided not to send and why.

---

Step 4 — Artifact Capture & Retention #

Every evaluation, whether it results in a notification or a suppressed entry, is written to a date-partitioned NDJSON log. This is the artifact an incident review pulls up to answer "did the system see this coming and choose not to alert, or did it simply never evaluate the finding at all" — those are very different failure modes and only a complete history log distinguishes them.

# /opt/alert-routing/history.py
# Requires: none beyond the standard library
from __future__ import annotations
import json
import os
from datetime import datetime, timezone
from pathlib import Path

RETENTION_DAYS = 180  # alert history retained longer than scoring artifacts for audit purposes


def append_alert_history(entries: list[dict]) -> Path:
    log_dir = Path(os.environ["ALERT_HISTORY_LOG_DIR"])
    log_dir.mkdir(parents=True, exist_ok=True)
    date_str = datetime.now(tz=timezone.utc).strftime("%Y-%m-%d")
    log_path = log_dir / f"{date_str}.ndjson"
    with log_path.open("a") as fh:
        for entry in entries:
            entry["logged_at"] = datetime.now(tz=timezone.utc).isoformat()
            fh.write(json.dumps(entry) + "\n")
    return log_path


def purge_expired(log_dir_path: str) -> int:
    """Delete NDJSON files older than RETENTION_DAYS. Run weekly via cron."""
    cutoff = datetime.now(tz=timezone.utc).timestamp() - RETENTION_DAYS * 86400
    removed = 0
    for f in Path(log_dir_path).glob("*.ndjson"):
        if f.stat().st_mtime < cutoff:
            f.unlink()
            removed += 1
    return removed

Retention for alert history is set longer than for the scoring artifacts themselves — 180 days versus the 90-day scoring retention seen elsewhere in this section — because compliance and incident-review questions about "were we alerted correctly" tend to surface months after the underlying scored data has already rolled off. Ship the purge job as its own weekly cron entry, separate from the scoring pipeline's cron, so a change to one schedule never accidentally deletes the other's artifacts.

# /etc/cron.d/alert-history-purge
0 5 * * 0   alert-user  python3 /opt/alert-routing/history.py --purge

---

Verification Checklist #

Confirm the threshold-and-routing layer is behaving correctly before trusting it in production:

1. Validate thresholds.json against its schema: python3 -m jsonschema --instance /opt/alert-routing/thresholds.json thresholds.schema.json.
2. Confirm tier ordering is exhaustive — every possible score in [0.0, 1.0] classifies into exactly one tier or None; write a unit test that sweeps 1,000 evenly spaced scores through classify_tier and asserts no exception.
3. Fire a synthetic breach and confirm exactly one notification lands, then fire the identical breach again inside the cooldown window and confirm zero additional notifications: check redis-cli TTL cooldown:<key> returns a positive value.
4. Confirm suppressed entries are still logged: grep '"fired": false' /var/log/site-health-alerts/$(date -u +%F).ndjson | wc -l should be non-zero on any day with cooldown suppression active.
5. Verify section overrides apply correctly by scoring a synthetic checkout finding at 0.68 and confirming it classifies as critical (override floor 0.70), not warning.
6. Confirm the purge job removed only files older than RETENTION_DAYS: find /var/log/site-health-alerts -name '*.ndjson' -mtime +180 should return nothing the day after a purge run.

---

Troubleshooting #

Every scoring run pages on-call, even for stable scores

Root cause: score_floor for the critical tier is set above the normal operating range of the composite score, so a healthy site is permanently classified as breaching.

# Compare current score distribution against the configured critical floor
python3 -c "
import json, pandas as pd
df = pd.read_parquet('/tmp/latest_scored.parquet')
floor = json.load(open('/opt/alert-routing/thresholds.json'))['tiers']['critical']['score_floor']
print(f'p50={df.score.median():.3f} p05={df.score.quantile(0.05):.3f} floor={floor}')
"

Fix: lower score_floor to sit below the historical P05 of a healthy period, or migrate that tier to a rolling-percentile baseline so the floor tracks the site's actual distribution automatically.

---

Alerts are silently missing for a known regression

Root cause: the finding's section value from the scoring artifact does not match any key in section_overrides, and the global default floor is too loose for that section.

# List distinct section values in the last scoring artifact vs configured overrides
python3 -c "
import json, pandas as pd
df = pd.read_parquet('/tmp/latest_scored.parquet')
overrides = json.load(open('/opt/alert-routing/thresholds.json'))['section_overrides'].keys()
print('scored sections:', sorted(df.section.unique()))
print('configured overrides:', sorted(overrides))
"

Fix: reconcile the section-naming convention between the scoring pipeline's output and thresholds.json — typically a hyphen-vs-underscore mismatch — and add the missing override.

---

Dedup key changes on every run, so the same incident re-alerts constantly

Root cause: the raw score or a timestamp was included in the string hashed to build dedup_key, so the key never repeats even though the underlying finding is the same ongoing issue.

# Inspect the fields actually hashed
grep -A3 "def dedup_key" /opt/alert-routing/evaluate.py

Fix: rebuild dedup_key from only metric, section, and tier — remove any score or timestamp component, then flush stale Redis keys with redis-cli --scan --pattern 'cooldown:*' | xargs redis-cli DEL to reset state cleanly.

---

Cooldown appears to expire early, causing duplicate pages

Root cause: the routing service host's system clock is not NTP-synced, or TZ was not exported before Redis TTLs were computed, so ex= receives a value computed against the wrong wall clock.

# Check clock sync status and confirm TZ is exported in the service environment
timedatectl status | grep "System clock synchronized"
systemctl show alert-routing.service -p Environment | grep TZ

Fix: enable chrony or systemd-timesyncd on the routing host, restart the service, and confirm TZ=UTC is present in the unit's environment file, not just the interactive shell.

---

A section override is defined but never applied

Root cause: classify_tier merges the override with dict.update, which requires the override key names to exactly match the tier's own key names (score_floor), and a typo (score_threshold) is silently ignored rather than raising an error.

# Diff override keys against the canonical tier schema keys
python3 -c "
import json
cfg = json.load(open('/opt/alert-routing/thresholds.json'))
valid_keys = set(cfg['tiers']['critical'].keys())
for section, tiers in cfg['section_overrides'].items():
    for tier, override in tiers.items():
        bad = set(override.keys()) - valid_keys
        if bad:
            print(f'{section}.{tier} has unknown keys: {bad}')
"

Fix: correct the key name in thresholds.json and add the schema validation step from Step 1 to CI so a future typo fails the pull request instead of shipping silently.

---

Digest channel never sends, even though info-tier findings accumulated all day

Root cause: digest_cron in routing.yaml uses a five-field cron expression evaluated in the routing service's local time rather than UTC, so it never fires at the expected wall-clock hour in production.

# Evaluate the next scheduled digest fire time against the server's actual TZ
python3 -c "
from croniter import croniter
from datetime import datetime, timezone
print(croniter('0 9 * * *', datetime.now(timezone.utc)).get_next(datetime))
"

Fix: confirm the digest scheduler process also runs with TZ=UTC exported, matching every other component in this pipeline, and restart the batching worker after the fix.

---

Related #

• Monitoring, Alerting & Remediation — parent section covering the full detect-to-remediate loop
• Setting Dynamic Alert Baselines with Rolling Percentiles — migrating a tier's static floor to a rolling-percentile baseline
• Suppressing Alert Noise During Deploy Windows — muting expected wobble during releases without weakening cooldowns
• Integrating Audit Alerts with Incident Management — wiring a routed alert into PagerDuty, Opsgenie, or Slack with auto-resolve

Should I start with static thresholds or rolling-percentile baselines?

Start static. A fixed score floor per severity tier is easy to reason about, easy to review in a pull request, and gives you a working alert pipeline on day one. Move to rolling-percentile baselines only after you have at least 30 days of stable scoring history to compute a meaningful baseline against, which is covered in Setting Dynamic Alert Baselines with Rolling Percentiles.

What should the dedup key be built from?

Hash the metric name, the affected URL segment, and the severity tier together, not the raw score value. Including the raw score in the key breaks deduplication because the score changes slightly on every scoring run, so a single ongoing incident would generate a new key — and a new alert — on every evaluation cycle. Keep the key composition to exactly those three fields and resist the urge to add more granularity.

How do cooldowns interact with deploy windows?

A cooldown suppresses repeat notifications for the same dedup key; it does not suppress the first notification during a known deploy. If you need to mute expected post-deploy wobble entirely, layer a deploy-window suppression rule on top of cooldowns, as described in Suppressing Alert Noise During Deploy Windows, rather than stretching the cooldown to cover every release.

Can one finding route to more than one channel?

Yes. The routing config accepts a list of channels per severity tier, so a critical breach can page the on-call rotation through an incident platform and simultaneously post to a Slack channel for visibility, without duplicating the dedup and cooldown logic per channel. The dedup key and cooldown state are computed once per finding and reused across every channel in that tier's list.