13 min read

Choosing Dashboard Alert Thresholds from Score History

Every alert threshold is a prediction about how often you want to be interrupted, and most of them are made by accident. Somebody picks a round number, ships it, and then loosens it every time the channel gets noisy — a sequence of locally reasonable decisions that converges on a threshold which fires for nothing. This page is part of Wiring Health Scores into Dashboards & Alerts and covers deriving the number from the score history you already have, and predicting the alert rate before anyone is paged.

Where a threshold sits in real history One score axis divided by the observed distribution of a healthy ninety-day period. The lowest band is below the observed p05, where a threshold fires only on genuinely unusual days. The next band spans p05 to p25, where a threshold fires on the worse end of normal. The next spans p25 to p75, the ordinary operating range, where a threshold would fire on most days. The highest band is above p75, where a threshold fires essentially always. worse score better score Below p05 fires on unusual days only p05 to p25 fires on a bad-but-normal day p25 to p75 fires most days — noise Above p75 fires always A threshold picked as a round number lands wherever it lands. Deriving it from the observed distribution of a period everyone agrees was healthy makes the alert rate a decision rather than a surprise.

Environment isolation and dependency declaration 

set -euo pipefail
export THR_HISTORY_TABLE="serving.scores_history"
export THR_BASELINE_DAYS=90            # the period the floors are derived from
export THR_INCIDENT_FILE="/opt/audit/config/incident_windows.json"
export THR_CRITICAL_PCTL=0.05          # critical floor sits below this
export THR_WARNING_PCTL=0.20           # warning floor sits near this
export THR_TARGET_PAGES_PER_MONTH=4    # the rate you are actually choosing
export THR_MIN_SECTION_DAYS=30         # below this, fall back to site-wide

THR_TARGET_PAGES_PER_MONTH is the parameter that makes this exercise honest. Everything else is arithmetic; this is the decision. Four critical pages a month is a very different operational posture from twenty, and picking a percentile without stating the rate it implies is how teams end up surprised by their own configuration.

Implementation 

#!/usr/bin/env python3
# /opt/audit/thresholds/derive.py
# Derive per-section severity floors from a known-healthy period, and
# report the alert rate each candidate floor would have produced.
from __future__ import annotations
import json
import os
import pandas as pd
import sqlalchemy as sa

DAYS = int(os.environ.get("THR_BASELINE_DAYS", "90"))
CRIT_P = float(os.environ.get("THR_CRITICAL_PCTL", "0.05"))
WARN_P = float(os.environ.get("THR_WARNING_PCTL", "0.20"))
MIN_DAYS = int(os.environ.get("THR_MIN_SECTION_DAYS", "30"))


def load_history(engine) -> pd.DataFrame:
    q = sa.text("""
        SELECT run_date, section, composite_score
        FROM serving.scores_history
        WHERE run_date >= CURRENT_DATE - :days
    """)
    return pd.read_sql(q, engine, params={"days": DAYS},
                       parse_dates=["run_date"])


def drop_incidents(df: pd.DataFrame, path: str) -> pd.DataFrame:
    """Remove dated incident windows. A floor derived from a period that
    contains an outage will not fire on the next one."""
    with open(path) as fh:
        windows = json.load(fh)
    mask = pd.Series(True, index=df.index)
    for w in windows:
        start, end = pd.Timestamp(w["start"]), pd.Timestamp(w["end"])
        hit = df["run_date"].between(start, end)
        if w.get("section"):
            hit &= df["section"] == w["section"]
        mask &= ~hit
    return df[mask]


def derive(df: pd.DataFrame) -> pd.DataFrame:
    rows = []
    site_crit = df["composite_score"].quantile(CRIT_P)
    site_warn = df["composite_score"].quantile(WARN_P)
    for section, g in df.groupby("section"):
        if g["run_date"].nunique() < MIN_DAYS:
            rows.append({"section": section, "critical_floor": site_crit,
                         "warning_floor": site_warn, "source": "site_fallback",
                         "days": g["run_date"].nunique()})
            continue
        crit = g["composite_score"].quantile(CRIT_P)
        warn = g["composite_score"].quantile(WARN_P)
        # What would this floor have done over the baseline period?
        fired = (g["composite_score"] < crit).sum()
        rows.append({"section": section, "critical_floor": crit,
                     "warning_floor": warn, "source": "section",
                     "days": g["run_date"].nunique(),
                     "would_have_fired": int(fired),
                     "pages_per_month": round(fired / (DAYS / 30.0), 1)})
    return pd.DataFrame(rows).sort_values("pages_per_month", ascending=False)

The parts that matter:

  • drop_incidents runs before any percentile is computed. It is the step teams skip, and skipping it is self-defeating: a floor derived from a window containing an outage sits below that outage, so a repeat of exactly the same failure passes silently.
  • Sections with fewer than MIN_DAYS of history fall back to the site-wide floor rather than getting a floor derived from a handful of days. A percentile over twelve observations is not a percentile, it is one of the twelve observations.
  • would_have_fired and pages_per_month are computed and returned, not logged as a side note. The output of this script is not a set of numbers, it is a set of numbers with their consequences attached.
Two ways a threshold gets its number Two columns. The derived column selects a period everyone agrees was healthy, excludes known incidents from it, computes the score percentiles over that period, sets the critical floor below the p05 and the warning floor near the p20, then predicts the resulting alert rate before shipping. The chosen column picks a round number that sounds right, ships it, waits for complaints, and adjusts until the channel is quiet, which converges on a threshold that fires for nothing. DERIVED FROM HISTORY Pick a known-healthy period 90 days everyone agrees was fine Exclude known incidents or the baseline includes them Compute p05, p20, p50 over that period, per section Predict the alert rate first before anything is shipped The rate is a decision CHOSEN AS A ROUND NUMBER Pick something that sounds right "alert below 80" — why 80? Ship it, wait for complaints the feedback loop is a human being Loosen until the channel is quiet each step is locally reasonable Converge on firing for nothing a threshold nobody trusts The rate is an accident

Verification and smoke test 

/opt/audit/.venv/bin/python3 -m audit.thresholds.derive --dry-run | tee /tmp/floors.tsv

# 1. No section is predicted to page more often than the target.
awk -F'\t' -v target="${THR_TARGET_PAGES_PER_MONTH}"   'NR>1 && $6+0 > target { print "OVER TARGET: " $1 " at " $6 "/month"; bad=1 }
   END { exit bad }' /tmp/floors.tsv   && echo "PASS: every section within the paging target"

# 2. Every floor sits below the section's own median.
awk -F'\t' 'NR>1 && $2+0 >= $7+0 { print "FLOOR ABOVE MEDIAN: " $1; bad=1 }
   END { exit bad }' /tmp/floors.tsv   && echo "PASS: no floor above its own median"

# 3. Diff against the live config before applying anything.
diff <(jq -S '.sections' "$THRESHOLD_CONFIG")      <(python3 -c 'import json,sys;print(json.dumps(json.load(sys.stdin),sort_keys=True))'         < /tmp/floors.json) || echo "review the diff above before applying"

Expected output is PASS: every section within the paging target and PASS: no floor above its own median, followed by a diff for review. The third step is the one that stops a derivation from being applied blind — a floor that has moved a long way since last quarter is either a real change in the site or a defect in the baseline period, and both deserve a human looking at them.

Failure modes

Three ways the derivation goes wrong Three rows. A baseline period containing an unremoved incident sets the floor below the incident, so a repeat of it never alerts. A single site-wide threshold applied to structurally different sections fires constantly on the slow one and never on the fast one. And a threshold derived once and never revisited drifts out of alignment as the site changes, usually becoming permanently silent. SYMPTOM ROOT CAUSE FIX A repeat of a known incident does not fire the alert at all The incident is in there it was inside the period the percentiles were computed over Exclude incident windows keep a dated list and subtract it before computing anything One section pages daily while another never pages at all One threshold, many sections structurally different populations sharing a single floor Derive per section each section against its own healthy-period distribution The alert stopped firing months ago and nobody noticed It was never revisited the site changed, the floor stayed where it was Recompute quarterly and diff the new floors against the live ones before applying Excluding incidents from the baseline is the step most often skipped, and it is self-defeating to skip: a floor derived from a period containing an outage is a floor that will not fire on the next one.

Every section falls back to the site-wide floor

The history table has fewer distinct run dates than THR_MIN_SECTION_DAYS, usually because the serving layer's history table was recently rebuilt and lost its backfill. Check SELECT COUNT(DISTINCT run_date) FROM serving.scores_history before assuming the sections are thin.

The derived floors are all far below anything that ever happens

The baseline period contains a long degradation rather than a discrete incident, so the p05 sits inside a bad month rather than below a good one. Incident windows catch outages; a slow drift needs a human judgement about which period counts as healthy. Plot the section's daily median across the window before trusting a percentile taken over it.

Applying the new floors silences an alert that was working

That is the expected outcome when the old floor was set by the loosening-until-quiet process and the new one is derived from history — but confirm the direction. If the new floor is lower than the old one and the old one was firing usefully, the baseline period included the very condition the alert exists to catch. Re-derive with that window excluded and compare again, and route the change through the same review as any other alert-routing configuration change.

FAQ

Why derive a threshold from history instead of picking a target score?

Because a target score is a statement about where you want the site to be, and a threshold is a statement about when you want to be interrupted. They are different questions with different right answers. A threshold set at an aspirational score fires every day until the aspiration is met, at which point it is muted and stops working entirely. Deriving from the observed distribution of a healthy period makes the alert rate an explicit choice.

Why must incident windows be excluded from the baseline?

Because percentiles computed over a period containing an outage sit below that outage. The resulting floor is, by construction, one that the incident would not have crossed — so an exact repeat of the same failure passes silently. Keep a dated list of known incident windows alongside the threshold config and subtract them before any percentile is computed; it is a small file and it is the difference between a working alert and a decorative one.

Should every section get its own threshold?

Every section with enough history, yes. Sections differ structurally — a checkout flow and a documentation archive have genuinely different score distributions — so a single floor either fires constantly on the lower one or never on the higher one. Sections with fewer than about thirty distinct run dates should fall back to the site-wide floor instead, because a percentile over a dozen observations is just one of those observations.

How often should thresholds be re-derived?

Quarterly, and after any event that changes the site materially — a migration, a redesign, a significant traffic-mix shift. Re-derivation should always be a diff against the live configuration rather than a direct overwrite: a floor that has moved a long way is either a real change in the site or a defect in the baseline period, and both need a person to look at them before they are applied.