Suppressing Alert Noise During Deploy Windows #

Every release produces a few minutes of expected metric wobble: cold caches inflate LCP, a freshly shipped bundle costs an extra parse-and-compile cycle on INP, and a rolling restart briefly thins the pool of warm backend workers. None of that is a regression, but a threshold evaluator configured under configuring alert thresholds and routing cannot tell the difference between "the release is still warming up" and "the release broke something" unless it is told a deploy is in progress. Paging on-call for the first is alert fatigue; failing to page for the second is a missed regression. This page builds the deploy-aware suppression layer that keeps both outcomes correct.

Environment Isolation and Dependency Declaration #

Pin the client version and declare the allowlist as an environment variable, not a hardcoded list buried in the evaluator. A shared allowlist means the same suppression rule applies whether the alert originates from the threshold evaluator or a one-off backfill job.

# /opt/audit/deploy_suppression — absolute working directory
export PYTHONPATH=/opt/audit/deploy_suppression
export SUPPRESSION_REDIS_DSN="redis://alerts-cache.internal:6379/2"
export SUPPRESSION_ALLOWLIST="lcp_p75,inp_p75,cls_p75"
export SUPPRESSION_MAX_TTL_SECONDS=1200
export SUPPRESSION_DEFAULT_TTL_SECONDS=600

# requirements.txt
redis==5.0.4
pydantic==2.7.1

set -euo pipefail
cd /opt/audit/deploy_suppression
python3 -m venv .venv
.venv/bin/pip install --quiet -r requirements.txt

SUPPRESSION_MAX_TTL_SECONDS is a hard ceiling the registration function enforces even if a deploy pipeline requests a longer window — it is the single control that stops a misconfigured CI job from suppressing alerts for hours.

Implementation #

The suppression layer has three moving parts wired into one script: registering a self-expiring window when a deploy starts, gating the threshold evaluator's decision against that window and the metric allowlist, and forcing a recheck the instant the window closes so a masked regression cannot hide past expiry.

#!/usr/bin/env python3
# /opt/audit/deploy_suppression/deploy_suppression.py
"""
Deploy-aware alert suppression.
Reads SUPPRESSION_* from environment; wraps a redis client with
native TTL expiry so windows self-clear without a cleanup job.
"""

import os
import time
import json
import redis

# ── 1. load config from environment, never from literals in code ────────────
DSN = os.environ["SUPPRESSION_REDIS_DSN"]
ALLOWLIST = set(os.environ["SUPPRESSION_ALLOWLIST"].split(","))
MAX_TTL = int(os.environ["SUPPRESSION_MAX_TTL_SECONDS"])
DEFAULT_TTL = int(os.environ["SUPPRESSION_DEFAULT_TTL_SECONDS"])

r = redis.Redis.from_url(DSN, decode_responses=True)


def _window_key(scope: str) -> str:
    # one key per scope (service or section), never one global flag —
    # this is what lets concurrent deploys suppress independently.
    return f"deploy_window:{scope}"


def register_deploy_window(deploy_id: str, scope: str, ttl_seconds: int | None = None) -> None:
    """Called by the deploy pipeline the instant a rollout starts."""
    ttl = min(ttl_seconds or DEFAULT_TTL, MAX_TTL)  # enforce the hard ceiling
    payload = json.dumps({"deploy_id": deploy_id, "scope": scope, "registered_at": time.time()})
    # SET ... EX gives us native, self-expiring suppression — no sweeper job,
    # no stale window left behind if the pipeline crashes after registering.
    r.set(_window_key(scope), payload, ex=ttl)
    # schedule the forced recheck exactly at expiry (see step 4 below)
    r.zadd("deploy_window_recheck", {scope: time.time() + ttl})


def evaluate_alert(scope: str, metric: str, breached: bool) -> dict:
    """
    Called by the threshold evaluator in place of a direct route-to-on-call
    decision. Returns a dict the router and the audit log both consume.
    """
    if not breached:
        return {"action": "none", "scope": scope, "metric": metric}

    window_raw = r.get(_window_key(scope))
    in_window = window_raw is not None
    suppressible = metric in ALLOWLIST

    if in_window and suppressible:
        window = json.loads(window_raw)
        # suppressed, but never silently dropped — the audit log is the point
        r.xadd("suppression_audit_log", {
            "deploy_id": window["deploy_id"],
            "scope": scope,
            "metric": metric,
            "action": "suppressed",
        })
        return {"action": "suppressed", "scope": scope, "metric": metric, "deploy_id": window["deploy_id"]}

    # either no active window, or the metric is a hard-failure signal that
    # is never suppressible — both cases route normally to on-call.
    return {"action": "route", "scope": scope, "metric": metric}


def run_forced_rechecks(evaluate_fn) -> None:
    """
    Poll the sorted set of scheduled expiries and, for any scope whose
    window just closed, immediately re-run evaluation instead of waiting
    for the next natural polling cycle.
    """
    now = time.time()
    due = r.zrangebyscore("deploy_window_recheck", 0, now)
    for scope in due:
        evaluate_fn(scope)  # caller re-evaluates the scope's current metrics
        r.zrem("deploy_window_recheck", scope)


if __name__ == "__main__":
    # example registration call — invoked by the deploy pipeline's post-start hook
    register_deploy_window(deploy_id="rel-2026.07.05-3", scope="checkout-section", ttl_seconds=600)

register_deploy_window is the only entry point that writes a window, evaluate_alert is the only entry point the threshold evaluator calls before routing, and run_forced_rechecks closes the gap a plain TTL leaves open — without it, a regression that started one second before expiry would sit unnoticed until the evaluator's next scheduled pass, which on a five-minute poll cycle is a five-minute blind spot right when a fresh deploy is most likely to be broken.

Verification and Smoke Test #

Run this after wiring the suppression layer into the evaluator, before trusting it against a real release.

set -euo pipefail
cd /opt/audit/deploy_suppression

# 1. register a short window for a synthetic scope
.venv/bin/python -c "
from deploy_suppression import register_deploy_window
register_deploy_window('smoke-test-deploy', 'smoke-test-scope', ttl_seconds=5)
print('PASS: window registered')
"

# 2. a suppressible metric breach inside the window must be suppressed
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'lcp_p75', breached=True)
assert result['action'] == 'suppressed', result
print('PASS: suppressible metric suppressed inside window')
"

# 3. a hard-failure metric must still route, even inside the same window
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'http_5xx_rate', breached=True)
assert result['action'] == 'route', result
print('PASS: hard-failure metric still routed inside window')
"

# 4. after TTL expiry, the same suppressible metric must route again
sleep 6
.venv/bin/python -c "
from deploy_suppression import evaluate_alert
result = evaluate_alert('smoke-test-scope', 'lcp_p75', breached=True)
assert result['action'] == 'route', result
print('PASS: suppression cleared after TTL expiry')
"

Expected output:

PASS: window registered
PASS: suppressible metric suppressed inside window
PASS: hard-failure metric still routed inside window
PASS: suppression cleared after TTL expiry

If step 3 ever prints suppressed instead of route, the allowlist has been misconfigured to include a hard-failure metric — treat that as a release-blocking config bug, not a tuning nit, because it means a real outage co-occurring with a deploy would go unpaged.

Failure Modes #

TTL set too long masks a genuine post-deploy regression

A window registered with ttl_seconds=3600 "to be safe" suppresses a real LCP regression for an hour instead of ten minutes. The forced recheck at expiry still fires, but by then the regression has been live — unpaged — for the full hour.

# diagnostic: list currently active windows and their remaining TTL
redis-cli -u "$SUPPRESSION_REDIS_DSN" --scan --pattern 'deploy_window:*' | \
  xargs -I{} redis-cli -u "$SUPPRESSION_REDIS_DSN" TTL {}
# fix: cap and shrink to the ceiling immediately
export SUPPRESSION_MAX_TTL_SECONDS=900
redis-cli -u "$SUPPRESSION_REDIS_DSN" EXPIRE deploy_window:checkout-section 900

Scope mismatch between the deploy pipeline and the alert router

The deploy pipeline registers a window under scope="checkout" while the threshold evaluator routes alerts keyed by scope="checkout-section". The strings never match, evaluate_alert never finds an active window, and on-call gets paged for expected wobble on every single release for that section.

# diagnostic: confirm the scope used at registration matches the router's scope taxonomy
redis-cli -u "$SUPPRESSION_REDIS_DSN" KEYS 'deploy_window:*'
grep -r "scope=" /opt/audit/deploy_suppression/deploy_hooks/*.sh
# fix: align both sides to the routing config's scope names, see
# configuring alert thresholds and routing for the canonical scope list

A hard-failure metric slips onto the suppressible allowlist

Someone adds http_5xx_rate to SUPPRESSION_ALLOWLIST to quiet a noisy rolling-restart blip, and the next real outage that happens to overlap a deploy window goes unpaged for the full TTL.

# diagnostic: print the live allowlist and flag any non-wobble metric
echo "$SUPPRESSION_ALLOWLIST" | tr ',' '\n' | grep -E '5xx|error_rate|crawl_failure' \
  && echo "FAIL: hard-failure metric on allowlist" || echo "PASS: allowlist clean"
# fix: remove it and redeploy the evaluator config
export SUPPRESSION_ALLOWLIST="lcp_p75,inp_p75,cls_p75"

FAQ #

How long should a deploy suppression window last?

Match the TTL to how long a release actually needs to warm caches and rebuild JS bundles — typically 10 to 20 minutes, not one fixed default across every service. Set ttl_seconds per scope from historical deploy telemetry rather than a single global constant, and keep it as short as the data supports; SUPPRESSION_MAX_TTL_SECONDS should always cap it well below "just in case" durations.

What happens if a real regression occurs during the suppression window?

Only metrics on the suppressible allowlist — LCP and INP wobble, in this build — are gated by the window at all. Hard-failure signals like 5xx rate, crawl failures, and 404 spikes are excluded from suppression entirely and keep routing to on-call, so a genuine outage still pages even in the middle of a deploy. This is the same reasoning behind tracking metric trends across release cycles: a metric that recovers on its own after a release is noise, one that doesn't is a regression, and the two must never share a suppression rule.

Should suppressed alerts be silently dropped?

No. Every suppressed evaluation is written to an audit log with the deploy_id, the matched window, the metric, and the evaluated value, so a post-deploy review can catch anything that stayed marginal after the window closed instead of the evidence simply disappearing. Treat the log the same way you'd treat any other alert-history retention requirement.

Can suppression windows overlap across concurrent deploys?

Yes, provided each window is keyed by its own scope rather than a single global suppression flag. Concurrent deploys to different sections get independent windows with independent TTLs, so one team's release never masks a regression in an unrelated section. This is also why the scope taxonomy has to match exactly what's used when setting dynamic alert baselines with rolling percentiles — both layers key off the same scope string, and drift between them reopens the false-page problem this page solves.

---

Related #

• Configuring Alert Thresholds and Routing — parent guide covering the full threshold and routing configuration this suppression layer plugs into
• Setting Dynamic Alert Baselines with Rolling Percentiles — the companion approach for adapting thresholds themselves rather than gating on a fixed window