14 min read

Suppressing Alert Noise During Deploy Windows

Every release produces a few minutes of expected metric wobble: cold caches inflate LCP, a freshly shipped bundle costs an extra parse-and-compile cycle on INP, and a rolling restart briefly thins the pool of warm backend workers. None of that is a regression, but a threshold evaluator configured under configuring alert thresholds and routing cannot tell the difference between "the release is still warming up" and "the release broke something" unless it is told a deploy is in progress. Paging on-call for the first is alert fatigue; failing to page for the second is a missed regression. This page builds the deploy-aware suppression layer that keeps both outcomes correct.

Deploy Window Suppression Flow A deploy event registers a TTL-bound suppression window keyed by scope. Alert evaluation checks the active window: suppressible metrics inside the window are logged and suppressed, while hard-failure metrics still route to on-call. When the TTL expires, a forced recheck feeds back into evaluation immediately. Deploy event scope + ttl_seconds Register window self-expiring key, TTL TTL expiry forced re-check Evaluate alert window active + metric? Suppress + log audit trail, no page Route to on-call hard-failure metric on expiry forced recheck in window, allowlisted not allowlisted

Environment Isolation and Dependency Declaration

Pin the client version and declare the allowlist as an environment variable, not a hardcoded list buried in the evaluator. A shared allowlist means the same suppression rule applies whether the alert originates from the threshold evaluator or a one-off backfill job.

# /opt/audit/deploy_suppression — absolute working directory
export PYTHONPATH=/opt/audit/deploy_suppression
export SUPPRESSION_REDIS_DSN="redis://alerts-cache.internal:6379/2"
export SUPPRESSION_ALLOWLIST="lcp_p75,inp_p75,cls_p75"
export SUPPRESSION_MAX_TTL_SECONDS=1200
export SUPPRESSION_DEFAULT_TTL_SECONDS=600
# requirements.txt
redis==5.0.4
pydantic==2.7.1

set -euo pipefail
cd /opt/audit/deploy_suppression
python3 -m venv .venv
.venv/bin/pip install --quiet -r requirements.txt

SUPPRESSION_MAX_TTL_SECONDS is a hard ceiling the registration function enforces even if a deploy pipeline requests a longer window — it is the single control that stops a misconfigured CI job from suppressing alerts for hours.

Implementation

The suppression layer has three moving parts wired into one script: registering a self-expiring window when a deploy starts, gating the threshold evaluator's decision against that window and the metric allowlist, and forcing a recheck the instant the window closes so a masked regression cannot hide past expiry.

What a deploy window may silence A question card asks whether the metric is on the suppressible allowlist while a window is active, with three outcomes. A wobble metric such as LCP or INP inside an active window is suppressed and logged with its deploy id. The same metric outside any window routes normally. A hard-failure metric such as the 5xx rate routes regardless, because a release is exactly when a real outage is most likely. Is this metric suppressible, and is a window active for its scope? on the allowlist, window active Suppress and log record the deploy_id, the window and the value on the allowlist, no window Route normally nothing about this evaluation is special not on the allowlist Route regardless a release is when an outage is most likely Adding a hard-failure metric to the allowlist to quiet a rolling-restart blip is the failure this design exists to prevent: the next real outage during a deploy goes unpaged. 
#!/usr/bin/env python3
# /opt/audit/deploy_suppression/deploy_suppression.py
"""
Deploy-aware alert suppression.
Reads SUPPRESSION_* from environment; wraps a redis client with
native TTL expiry so windows self-clear without a cleanup job.
"""

import os
import time
import json
import redis

# ── 1. load config from environment, never from literals in code ────────────
DSN = os.environ["SUPPRESSION_REDIS_DSN"]
ALLOWLIST = set(os.environ["SUPPRESSION_ALLOWLIST"].split(","))
MAX_TTL = int(os.environ["SUPPRESSION_MAX_TTL_SECONDS"])
DEFAULT_TTL = int(os.environ["SUPPRESSION_DEFAULT_TTL_SECONDS"])

r = redis.Redis.from_url(DSN, decode_responses=True)


def _window_key(scope: str) -> str:
    # one key per scope (service or section), never one global flag —
    # this is what lets concurrent deploys suppress independently.
    return f"deploy_window:{scope}"


def register_deploy_window(deploy_id: str, scope: str, ttl_seconds: int | None = None) -> None:
    """Called by the deploy pipeline the instant a rollout starts."""
    ttl = min(ttl_seconds or DEFAULT_TTL, MAX_TTL)  # enforce the hard ceiling
    payload = json.dumps({"deploy_id": deploy_id, "scope": scope, "registered_at": time.time()})
    # SET ... EX gives us native, self-expiring suppression — no sweeper job,
    # no stale window left behind if the pipeline crashes after registering.
    r.set(_window_key(scope), payload, ex=ttl)
    # schedule the forced recheck exactly at expiry (see step 4 below)
    r.zadd("deploy_window_recheck", {scope: time.time() + ttl})


def evaluate_alert(scope: str, metric: str, breached: bool) -> dict:
    """
    Called by the threshold evaluator in place of a direct route-to-on-call
    decision. Returns a dict the router and the audit log both consume.
    """
    if not breached:
        return {"action": "none", "scope": scope, "metric": metric}

    window_raw = r.get(_window_key(scope))
    in_window = window_raw is not None
    suppressible = metric in ALLOWLIST

    if in_window and suppressible:
        window = json.loads(window_raw)
        # suppressed, but never silently dropped — the audit log is the point
        r.xadd("suppression_audit_log", {
            "deploy_id": window["deploy_id"],
            "scope": scope,
            "metric": metric,
            "action": "suppressed",
        })
        return {"action": "suppressed", "scope": scope, "metric": metric, "deploy_id": window["deploy_id"]}

    # either no active window, or the metric is a hard-failure signal that
    # is never suppressible — both cases route normally to on-call.
    return {"action": "route", "scope": scope, "metric": metric}


def run_forced_rechecks(evaluate_fn) -> None:
    """
    Poll the sorted set of scheduled expiries and, for any scope whose
    window just closed, immediately re-run evaluation instead of waiting
    for the next natural polling cycle.
    """
    now = time.time()
    due = r.zrangebyscore("deploy_window_recheck", 0, now)
    for scope in due:
        evaluate_fn(scope)  # caller re-evaluates the scope's current metrics
        r.zrem("deploy_window_recheck", scope)


if __name__ == "__main__":
    # example registration call — invoked by the deploy pipeline's post-start hook
    register_deploy_window(deploy_id="rel-2026.07.05-3", scope="checkout-section", ttl_seconds=600)

register_deploy_window is the only entry point that writes a window, evaluate_alert is the only entry point the threshold evaluator calls before routing, and run_forced_rechecks closes the gap a plain TTL leaves open — without it, a regression that started one second before expiry would sit unnoticed until the evaluator's next scheduled pass, which on a five-minute poll cycle is a five-minute blind spot right when a fresh deploy is most likely to be broken.

Verification and Smoke Test

Run this after wiring the suppression layer into the evaluator, before trusting it against a real release.

set -euo pipefail
cd /opt/audit/deploy_suppression

# 1. register a short window for a synthetic scope
.venv/bin/python -c "
from deploy_suppression import register_deploy_window
register_deploy_window('smoke-test-deploy', 'smoke-test-scope', ttl_seconds=5)
print('PASS: window registered')
"

# 2. a suppressible metric breach inside the window must be suppressed
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'lcp_p75', breached=True)
assert result['action'] == 'suppressed', result
print('PASS: suppressible metric suppressed inside window')
"

# 3. a hard-failure metric must still route, even inside the same window
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'http_5xx_rate', breached=True)
assert result['action'] == 'route', result
print('PASS: hard-failure metric still routed inside window')
"

# 4. after TTL expiry, the same suppressible metric must route again
sleep 6
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'lcp_p75', breached=True)
assert result['action'] == 'route', result
print('PASS: suppression cleared after TTL expiry')
"

Expected output:

PASS: window registered
PASS: suppressible metric suppressed inside window
PASS: hard-failure metric still routed inside window
PASS: suppression cleared after TTL expiry

If step 3 ever prints suppressed instead of route, the allowlist has been misconfigured to include a hard-failure metric — treat that as a release-blocking config bug, not a tuning nit, because it means a real outage co-occurring with a deploy would go unpaged.

Failure Modes

TTL set too long masks a genuine post-deploy regression

Three ways suppression goes too far Three rows. A window registered with an hour-long TTL to be safe suppresses a genuine post-deploy regression for fifty minutes longer than the release needed, which is why the registration function enforces a hard ceiling. A scope string that differs between the deploy pipeline and the alert router means the window matches nothing and suppresses nothing, so the two read the same shared constant. A hard-failure metric added to the allowlist silences the one class of alert that must always page. SYMPTOM ROOT CAUSE FIX A real regression is silent for the better part of an hour after a release The TTL was set generously an hour "to be safe" instead of the ten minutes actually needed Enforce a ceiling the registration function caps the TTL regardless of the request Nothing is suppressed the deploy still produces the usual burst of wobble alerts Scope strings disagree the pipeline registers checkout, the router keys checkout-section Share one constant both sides read the same scope value from one place An outage goes unpaged because it happened to start during a release A hard-failure metric was added to the allowlist to quiet a restart blip Keep the allowlist narrow wobble metrics only — treat a 5xx rate on it as an incident Every suppressed evaluation is still written to the audit log with its deploy id and value, so a postmortem can always answer "did we know, and did we choose not to page".

A window registered with ttl_seconds=3600 "to be safe" suppresses a real LCP regression for an hour instead of ten minutes. The forced recheck at expiry still fires, but by then the regression has been live — unpaged — for the full hour.

# diagnostic: list currently active windows and their remaining TTL
redis-cli -u "$SUPPRESSION_REDIS_DSN" --scan --pattern 'deploy_window:*' | \
  xargs -I{} redis-cli -u "$SUPPRESSION_REDIS_DSN" TTL {}
# fix: cap and shrink to the ceiling immediately
export SUPPRESSION_MAX_TTL_SECONDS=900
redis-cli -u "$SUPPRESSION_REDIS_DSN" EXPIRE deploy_window:checkout-section 900

Scope mismatch between the deploy pipeline and the alert router

The deploy pipeline registers a window under scope="checkout" while the threshold evaluator routes alerts keyed by scope="checkout-section". The strings never match, evaluate_alert never finds an active window, and on-call gets paged for expected wobble on every single release for that section.

# diagnostic: confirm the scope used at registration matches the router's scope taxonomy
redis-cli -u "$SUPPRESSION_REDIS_DSN" KEYS 'deploy_window:*'
grep -r "scope=" /opt/audit/deploy_suppression/deploy_hooks/*.sh
# fix: align both sides to the routing config's scope names, see
# configuring alert thresholds and routing for the canonical scope list

A hard-failure metric slips onto the suppressible allowlist

Someone adds http_5xx_rate to SUPPRESSION_ALLOWLIST to quiet a noisy rolling-restart blip, and the next real outage that happens to overlap a deploy window goes unpaged for the full TTL.

# diagnostic: print the live allowlist and flag any non-wobble metric
echo "$SUPPRESSION_ALLOWLIST" | tr ',' '\n' | grep -E '5xx|error_rate|crawl_failure' \
  && echo "FAIL: hard-failure metric on allowlist" || echo "PASS: allowlist clean"
# fix: remove it and redeploy the evaluator config
export SUPPRESSION_ALLOWLIST="lcp_p75,inp_p75,cls_p75"

FAQ

How long should a deploy suppression window last?

Match the TTL to how long a release actually needs to warm caches and rebuild JS bundles — typically 10 to 20 minutes, not one fixed default across every service. Set ttl_seconds per scope from historical deploy telemetry rather than a single global constant, and keep it as short as the data supports; SUPPRESSION_MAX_TTL_SECONDS should always cap it well below "just in case" durations.

What happens if a real regression occurs during the suppression window?

Only metrics on the suppressible allowlist — LCP and INP wobble, in this build — are gated by the window at all. Hard-failure signals like 5xx rate, crawl failures, and 404 spikes are excluded from suppression entirely and keep routing to on-call, so a genuine outage still pages even in the middle of a deploy. This is the same reasoning behind tracking metric trends across release cycles: a metric that recovers on its own after a release is noise, one that doesn't is a regression, and the two must never share a suppression rule.

Should suppressed alerts be silently dropped?

No. Every suppressed evaluation is written to an audit log with the deploy_id, the matched window, the metric, and the evaluated value, so a post-deploy review can catch anything that stayed marginal after the window closed instead of the evidence simply disappearing. Treat the log the same way you'd treat any other alert-history retention requirement.

Can suppression windows overlap across concurrent deploys?

Yes, provided each window is keyed by its own scope rather than a single global suppression flag. Concurrent deploys to different sections get independent windows with independent TTLs, so one team's release never masks a regression in an unrelated section. This is also why the scope taxonomy has to match exactly what's used when setting dynamic alert baselines with rolling percentiles — both layers key off the same scope string, and drift between them reopens the false-page problem this page solves.