12 min read

Assigning Severity Labels to Audit Findings

A finding taxonomy without a severity rubric is just a list of complaints. Every audit run surfaces dozens to thousands of rows — broken canonicals, orphaned redirects, missing hreflang, blown Core Web Vitals — and without a deterministic way to turn each row into a P1–P4 label, triage becomes whoever shouts loudest in the incident channel. This page is part of Mapping Audit Findings to Remediation Workflows and shows the exact scoring function — impact × reach × confidence — that a finding taxonomy needs before it can route anything automatically.

The rubric has to survive two properties or it is worthless in production: the same finding scored twice must produce the same label, and two engineers scoring the same finding independently must agree. Both properties come from binding every input to an observable condition instead of a feeling, which is the core of the implementation below.

Finding Severity Scoring Flow Diagram showing a raw finding rated on three axes — impact, reach, and confidence — feeding a multiplication step that produces a composite score, which is then mapped through band boundaries to a P1, P2, P3, or P4 label. Raw Finding taxonomy row from crawl report Score 3 axes impact (1–5) reach (0–1) confidence (0–1) bound to observable taxonomy conditions multiply Composite 0.0 – 5.0 score band Severity Label P1 ≥ 3.00 — page down P2 ≥ 1.50 — this week P3 ≥ 0.60 — this sprint P4 < 0.60 — backlog ties broken by reach, then impact

Implementation

The scoring function below takes a finding row (already carrying an impact rating, a reach fraction computed against the current crawl snapshot, and a confidence value) and returns a stable severity label. It is deliberately a pure function — no I/O, no randomness — so it can be unit-tested and re-run against historical finding sets to check for drift.

#!/usr/bin/env python3
# /opt/audit/severity/score_findings.py
"""
Deterministic severity rubric: severity = impact x reach x confidence.
Consumes finding rows emitted by the taxonomy mapper and returns each
row annotated with a composite score and a P1-P4 label.
"""

from dataclasses import dataclass
from typing import Literal

Severity = Literal["P1", "P2", "P3", "P4"]

# ── 1. band boundaries, pinned here and nowhere else ─────────────────────────
# composite = impact(1-5) x reach(0-1) x confidence(0-1); max possible = 5.0
BANDS: list[tuple[float, Severity]] = [
    (3.00, "P1"),   # page-down or index-blocking, affecting most of the site
    (1.50, "P2"),   # material user or crawl impact, must ship this week
    (0.60, "P3"),   # real but contained, fits in the normal sprint queue
    (0.00, "P4"),   # cosmetic or single-URL, backlog only
]

@dataclass(frozen=True)
class Finding:
    finding_code: str
    url_count: int          # URLs this finding instance touches
    total_indexable: int     # indexable URLs in the current crawl snapshot
    impact: int              # 1-5, bound to a taxonomy condition (see below)
    confidence: float        # 0.0-1.0, how sure the detector is this is real

    @property
    def reach(self) -> float:
        # reach is always computed against total indexable URLs, never
        # against total crawled URLs, so noindex/blocked pages don't dilute it
        if self.total_indexable == 0:
            return 0.0
        return min(self.url_count / self.total_indexable, 1.0)

    @property
    def composite(self) -> float:
        return round(self.impact * self.reach * self.confidence, 4)


def label_for(composite: float) -> Severity:
    for floor, label in BANDS:
        if composite >= floor:
            return label
    return "P4"  # unreachable given BANDS ends at 0.00, kept for clarity


def rank_key(f: Finding) -> tuple:
    # deterministic tie-break: composite desc, then reach desc, then impact desc,
    # then finding_code asc as the final, always-unique tiebreaker
    return (-f.composite, -f.reach, -f.impact, f.finding_code)


def score_all(findings: list[Finding]) -> list[dict]:
    scored = []
    for f in findings:
        scored.append({
            "finding_code": f.finding_code,
            "impact": f.impact,
            "reach": round(f.reach, 4),
            "confidence": f.confidence,
            "composite": f.composite,
            "severity": label_for(f.composite),
        })
    # sort the finding set itself so downstream routing consumes a stable queue
    findings_sorted = sorted(findings, key=rank_key)
    order = {f.finding_code: i for i, f in enumerate(findings_sorted)}
    for row in scored:
        row["queue_position"] = order[row["finding_code"]]
    return sorted(scored, key=lambda r: r["queue_position"])


if __name__ == "__main__":
    sample = [
        Finding("noindex-on-canonical", url_count=1400, total_indexable=12000,
                impact=5, confidence=0.95),
        Finding("missing-hreflang-alt", url_count=80, total_indexable=12000,
                impact=2, confidence=0.80),
        Finding("redirect-chain-3plus", url_count=340, total_indexable=12000,
                impact=3, confidence=0.90),
    ]
    for row in score_all(sample):
        print(row)

Line-by-line, the parts that matter for correctness: reach is a property computed on the fly against total_indexable, never stored as a raw count, so the same finding always reads the crawl snapshot it was scored against. BANDS is a single ordered list checked top-down, so there is exactly one code path that decides a label — no duplicated if logic to drift out of sync between a Python service and a downstream dashboard. rank_key exists because two findings can legitimately land in the same band; without an explicit, total ordering, dict iteration order or scan order becomes the tiebreaker, which is not reproducible across Python versions or process restarts.

impact itself must be bound to the finding taxonomy referenced in Mapping Audit Findings to Remediation Workflows — for example, impact=5 is reserved for "blocks indexing of a canonical URL" and impact=2 for "missing a non-required annotation." Keeping that mapping in the taxonomy file rather than inline in this script is what lets two reviewers agree on a score.

Verification and Smoke Test

Run this after any change to BANDS, the taxonomy's impact bindings, or the reach calculation, before the output is wired into remediation routing.

set -euo pipefail
cd /opt/audit/severity

# 1. run the scorer twice against the same fixture and diff the output
.venv/bin/python score_findings.py > /tmp/run1.txt
.venv/bin/python score_findings.py > /tmp/run2.txt
diff /tmp/run1.txt /tmp/run2.txt && echo "PASS: identical output across re-runs"

# 2. assert every band boundary is covered by a unit test
.venv/bin/python -m pytest tests/test_bands.py -q

# 3. assert queue order is a total order (no two findings share a queue_position)
.venv/bin/python -c "
from score_findings import Finding, score_all
sample = [
    Finding('a', 100, 1000, impact=3, confidence=0.9),
    Finding('b', 100, 1000, impact=3, confidence=0.9),
]
rows = score_all(sample)
positions = [r['queue_position'] for r in rows]
assert positions == sorted(set(positions)), 'duplicate queue positions'
print('PASS: total order holds under a composite tie')
"

Expected output:

PASS: identical output across re-runs
2 passed in 0.04s
PASS: total order holds under a composite tie

If the re-run diff is non-empty, something in the pipeline is non-deterministic — usually reach being computed against a crawl snapshot that changed mid-run rather than the rubric itself.

Failure Modes

Impact scores are effectively a vibe, not a rating

If two reviewers give the same finding different impact scores, the taxonomy's condition text is too loose. This surfaces as label churn that has nothing to do with the underlying issue changing.

# diagnostic: find findings whose impact disagrees across two reviewer passes
.venv/bin/python -c "
import json
a = json.load(open('review_pass_a.json'))
b = json.load(open('review_pass_b.json'))
for code in a:
    if a[code]['impact'] != b[code]['impact']:
        print(code, a[code]['impact'], 'vs', b[code]['impact'])
"
# fix: tighten the taxonomy condition text for every finding_code printed above

Two findings tie on composite score and route in an unstable order

Without the rank_key tiebreaker, findings with equal impact x reach x confidence can flip order between runs depending on dict or scan order, which makes on-call queues unpredictable.

# diagnostic: look for composite ties in the current finding set
.venv/bin/python -c "
from score_findings import score_all
import json
findings = json.load(open('current_findings.json'))
rows = score_all(findings)
seen = {}
for r in rows:
    seen.setdefault(r['composite'], []).append(r['finding_code'])
for comp, codes in seen.items():
    if len(codes) > 1:
        print(f'tie at {comp}:', codes)
"
# fix: confirm rank_key is applied before the queue is persisted or displayed

Severity label drifts between consecutive audit runs on an unchanged finding

This is almost always a stale or shifted total_indexable denominator, not a real severity change, especially right after a large section is deindexed or a new site section launches.

# diagnostic: compare total_indexable between the two runs' crawl snapshots
.venv/bin/python -c "
import json
r1 = json.load(open('snapshot_2026-06-28.json'))
r2 = json.load(open('snapshot_2026-07-05.json'))
print('prior total_indexable:', r1['total_indexable'])
print('current total_indexable:', r2['total_indexable'])
"
# fix: pin the snapshot id used for reach on the finding record itself,
# and re-score history against the pinned snapshot before comparing labels

FAQ

Why multiply impact, reach, and confidence instead of adding them?

Addition lets a high score on one axis compensate for a near-zero score on another, which produces false P1s for findings that are severe but affect one URL, or widespread but trivial. Multiplication forces all three axes to be non-trivial before a finding can reach the top band, which matches how blast radius actually behaves.

Who decides the impact score, and how do we stop it from being subjective?

Impact should never be a free-text guess. Bind each impact value (1-5) to an observable condition in the finding taxonomy, such as "blocks indexing" or "degrades a Core Web Vital past the Poor threshold." Two reviewers scoring the same finding against the bound conditions should produce the same number; if they do not, the condition text is too vague and needs tightening, not the reviewer.

How is this different from the risk-scoring framework for technical debt?

The risk-scoring framework ranks accumulated technical debt over a longer horizon and factors in cost-to-fix. This rubric scores a single fresh finding at triage time, purely to decide how fast it needs a human, and feeds directly into the routing step described in prioritizing critical vs non-critical site errors.

What happens when a finding's severity label changes between audit runs?

Label drift on an unchanged finding almost always means reach was recomputed against a different crawl snapshot, not that the underlying issue changed. Pin the crawl snapshot ID used for reach calculation into the finding record so a label change can be traced to either a genuine severity shift or a stale-input artifact.