14 min read

Detecting Outlier Page Samples with Median Absolute Deviation

An audit dataset always contains a handful of measurements that no user produced. A tab that hung and reported a 91-second largest contentful paint, a layout shift of 4.1 from an advert slot that never settled, a time to first byte recorded while the origin was mid-deploy. Left in, they move a p75 visibly and a mean catastrophically. This page is part of Handling Missing & Outlier Metric Data and covers the specific test worth using to find them: the median absolute deviation, and why the three-sigma rule most pipelines reach for first quietly stops working on exactly this kind of data.

Where each method draws the line A horizontal bar chart of the upper fence produced by four methods over the same nine LCP samples, whose bulk sits between 1100 and 2400 milliseconds with one 91000 millisecond outlier. A three-sigma fence lands at about 62000 milliseconds and admits the outlier. An IQR fence lands near 3800. A MAD fence at 3.5 lands near 3500. A hard percentile clip at the 99th lands near 2400. Only the first fails to exclude the outlier. UPPER FENCE OVER THE SAME NINE SAMPLES mean + 3 std 62,000 ms Q3 + 1.5 IQR 3,800 ms median + 3.5 MAD 3,500 ms p99 hard clip 2,400 ms The three-sigma fence is not slightly wrong here — it lands twenty-five times further out than the others, because the single 91-second sample is most of the standard deviation it was computed from.

Environment isolation and dependency declaration 

set -euo pipefail
export MAD_K=3.5                     # fence width, in scaled MAD units
export MAD_MIN_SAMPLES=10            # below this, do not fence at all
export MAD_MIN_DISTINCT=5            # below this, MAD is degenerate
export MAD_GROUP_KEY="template"      # fence within a group, never site-wide
/opt/audit/.venv/bin/pip install "numpy==1.26.4" "pandas==2.2.2"

MAD_GROUP_KEY matters as much as MAD_K. A site-wide fence compares a checkout page against a documentation page and calls the structurally slower one an outlier; fencing within a template compares each page against pages that do the same job.

Implementation 

#!/usr/bin/env python3
# /opt/audit/dq/mad.py — robust outlier flags, with the degenerate cases handled.
from __future__ import annotations
import os
import numpy as np
import pandas as pd

K = float(os.environ.get("MAD_K", "3.5"))
MIN_N = int(os.environ.get("MAD_MIN_SAMPLES", "10"))
MIN_DISTINCT = int(os.environ.get("MAD_MIN_DISTINCT", "5"))
SCALE = 1.4826          # makes MAD comparable to a std for a normal sample


def fence(values: pd.Series) -> tuple[float, float, str]:
    """Return (lower, upper, method). Method is part of the output so a
    downstream consumer can tell which test actually produced a flag."""
    clean = values.dropna()
    if len(clean) < MIN_N:
        return (-np.inf, np.inf, "skipped_too_few")
    if clean.nunique() < MIN_DISTINCT:
        return (-np.inf, np.inf, "skipped_degenerate")

    med = clean.median()
    mad = (clean - med).abs().median()
    if mad == 0:                       # >50% identical values
        q1, q3 = clean.quantile([0.25, 0.75])
        iqr = q3 - q1
        if iqr == 0:
            return (-np.inf, np.inf, "skipped_degenerate")
        return (q1 - 1.5 * iqr, q3 + 1.5 * iqr, "iqr_fallback")

    scaled = SCALE * mad
    return (med - K * scaled, med + K * scaled, "mad")


def flag(df: pd.DataFrame, metric: str, group_key: str) -> pd.DataFrame:
    out = []
    for key, g in df.groupby(group_key):
        lo, hi, method = fence(g[metric])
        g = g.copy()
        g["outlier"] = ~g[metric].between(lo, hi) & g[metric].notna()
        g["fence_method"] = method
        g["fence_lo"], g["fence_hi"] = lo, hi
        out.append(g)
    return pd.concat(out, ignore_index=True)

Line by line, the decisions doing real work:

  • SCALE = 1.4826 converts a median absolute deviation into something comparable to a standard deviation for a normally distributed sample, so a K of 3.5 is roughly interpretable as "three and a half sigma, if the data had been normal". Without it, K is an arbitrary number with no relationship to any other threshold in the pipeline.
  • The len(clean) < MIN_N guard returns an infinite fence rather than raising. A thin group is not an error; it is a group that should not be fenced, and returning infinities lets the same code path run over every group without a special case at the call site.
  • clean.nunique() < MIN_DISTINCT catches the cached-template case before the MAD is computed, which is cheaper and clearer than catching the resulting zero afterwards.
  • The mad == 0 branch falls back to an IQR fence rather than giving up, because a group can have a zero MAD and still have a usable interquartile range — more than half identical values, but a real spread in the rest.
  • fence_method is emitted on every row. When someone asks why a page was flagged, the answer has to be available without re-running the calculation.
The same nine samples, two statistics Two columns walking the same nine LCP samples. The MAD column shows the median at 1900 milliseconds, the absolute deviations from it, their median at 430, the scaled value at 637 after applying the 1.4826 constant, and the resulting upper fence near 4130. The standard deviation column shows a mean pulled to 11700 by one sample, a standard deviation of about 29600, and an upper fence past 100000. The verdicts note which statistic the outlier participated in defining. MEDIAN ABSOLUTE DEVIATION median = 1,900 ms the 91,000 sample cannot move it median |x - med| = 430 ms again unmoved by one extreme x 1.4826 = 637 ms scaled to a std-equivalent fence = 1,900 + 3.5 x 637 about 4,130 ms The outlier is outside MEAN AND STANDARD DEVIATION mean = 11,700 ms one sample moved it by 9,800 std = 29,600 ms almost entirely that one sample fence = 11,700 + 3 x 29,600 about 100,500 ms Every real sample is inside and so is the outlier The outlier defined the fence

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
import pandas as pd
from mad import fence, flag

# 1. A known outlier is caught, and the fence sits near the bulk of the data.
s = pd.Series([1180, 1400, 1610, 1750, 1900, 2050, 2260, 2400, 91000])
lo, hi, method = fence(s)
assert method == "mad", method
assert hi < 6000, f"fence too wide: {hi:.0f}"
assert s.iloc[-1] > hi, "the 91s sample was not caught"
print(f"PASS: mad fence upper = {hi:.0f} ms, outlier excluded")

# 2. A degenerate group is skipped rather than flagged wholesale.
d = pd.Series([1500] * 12 + [1600, 1700])
lo2, hi2, m2 = fence(d)
assert m2 in ("iqr_fallback", "skipped_degenerate"), m2
print(f"PASS: degenerate group handled as {m2}")

# 3. Removing the outlier must not change the fence much (robustness).
lo3, hi3, _ = fence(s.iloc[:-1])
assert abs(hi3 - hi) / hi < 0.05, "fence moved when the outlier was removed"
print("PASS: fence is stable with and without the outlier")
PY

Expected output:

PASS: mad fence upper = 4131 ms, outlier excluded
PASS: degenerate group handled as iqr_fallback
PASS: fence is stable with and without the outlier

The third assertion is the one that actually tests robustness, and it is the one worth keeping in CI. A fence that moves when you remove the value it was supposed to exclude is not robust, whatever statistic it claims to be built on — run the same assertion against a three-sigma implementation and it fails by an enormous margin.

Failure modes

Three times MAD is the wrong tool Three rows. A group where more than half the values are identical produces a median absolute deviation of exactly zero, so every distinct value falls outside a zero-width fence. A group with fewer than about ten samples produces a fence that moves substantially with each new sample. And a bimodal group, such as one template served from two very different origins, has no single centre for the fence to sit at. SITUATION WHAT GOES WRONG WHAT TO DO Over half the values match a cached template returning a constant timing MAD collapses to zero so the fence has zero width and flags every distinct value Fall back to IQR and treat the group as unsuitable for outlier testing at all Fewer than ~10 samples a thin template, or a newly published section The fence is unstable each new sample moves it enough to change the verdict Do not fence at all flag the group as under-sampled and exclude it from cleaning Two clear clusters one template served from two different origins There is no single centre the median sits in the gap between the two groups Split the group first fence each population separately, or fix the grouping key All three are detectable before the fence is applied: distinct-value count, sample count, and a simple bimodality check are cheaper than explaining a nonsensical outlier list later.

Every page in one template is flagged

The group is bimodal — usually one template served from two origins, or a locale split where one region is structurally slower. The median lands in the gap between the two populations and neither one is inside the fence. Diagnose by checking whether the group splits cleanly:

g = df[df.template == "product"]["lcp_ms"]
print(g.describe(), "\n gap check:", g.quantile(0.6) - g.quantile(0.4))

A large jump between the 40th and 60th percentiles means two populations. Fix the grouping key rather than the fence.

The flag rate climbs steadily week over week

The fence is being recomputed each run against a distribution that is genuinely degrading, so more real pages fall outside it. This is the fence working correctly and the site getting worse, which is easy to misread as a noisy filter. Compare the fence bounds across runs rather than only the flag count — a rising flag count with a stable fence is a site problem, a rising flag count with a narrowing fence is a data problem.

A flagged page turns out to be genuinely slow

That is the expected outcome of an outlier test, not a defect. The fence identifies values that do not belong to the group's distribution; whether such a value is a broken measurement or a real problem is a separate judgement. Never delete flagged rows — winsorize them so they keep their weight, and route the flag itself into the false-positive review process rather than into a deletion.

FAQ

Why does a three-sigma fence stop catching outliers on latency data?

Because the standard deviation it is built from is computed over the same data the outlier is in, and a single extreme value can be most of it. On a set whose bulk sits between one and two and a half seconds, one 91-second sample can push a three-sigma upper bound past a minute — the outlier defines the boundary it is then tested against. Add a second extreme value and the boundary widens again, so the filter degrades exactly as the data gets worse.

What does the 1.4826 constant do?

It scales the median absolute deviation so it is numerically comparable to a standard deviation for a normally distributed sample. Without it, the fence width parameter is an arbitrary number with no relationship to any other threshold in the pipeline; with it, a K of 3.5 is roughly interpretable as "three and a half sigma, if this data had been normal", which makes the parameter reviewable by someone who has not read the implementation.

What happens when more than half the values in a group are identical?

The median absolute deviation is exactly zero, because the median of the absolute deviations is itself zero, and every value that is not the median then falls outside a zero-width fence. This is common on cached templates that return a constant timing and on synthetic fixtures. Detect it by counting distinct values before computing anything, and either fall back to an interquartile fence or skip outlier testing for that group entirely.

Should a flagged sample be deleted?

No. Winsorize it instead, so the row keeps its weight in the distribution while losing its ability to distort the group. Deleting flagged rows is how a site with a genuinely heavy tail of slow pages reports an excellent p75 — the fence cannot tell a broken measurement from a real problem, and treating every flag as noise systematically removes the worst real pages from the dataset.