17 min read

Risk Scoring Frameworks for Technical Debt

Without a consistent scoring model, teams waste sprint cycles on cosmetic issues while indexability loss and LCP regressions compound undetected. This workflow — part of the broader Technical Audit Fundamentals & Scope Mapping practice — converts raw crawl data into a reproducible composite score that drives automated triage, not just reports.

Risk Scoring Pipeline Five-stage data-flow diagram showing how raw crawl data passes through ingestion, normalisation, weight assignment, composite scoring, and alert routing stages. Ingest & Validate Normalise Metrics Apply Weights Composite Risk Score Route Alerts Jira / PagerDuty Score bands: Critical ≥85 High 65–84 Medium 40–64 Low <40

Prerequisites & Environment Setup

The scoring engine depends on Python 3.11+, pandas 2.2, numpy 1.26, and BigQuery client libraries. Pin versions in a lockfile so CI runs match local results.

# requirements.txt
pandas==2.2.2
numpy==1.26.4
google-cloud-bigquery==3.21.0
requests==2.32.2
pyyaml==6.0.1

Required environment variables — export these before running any pipeline step:

export SCORING_BASE_URL="https://example.com"
export SCORING_SITEMAP_URL="https://example.com/sitemap.xml"
export BQ_PROJECT="your-gcp-project"
export BQ_DATASET="audit_metrics"
export JIRA_BASE_URL="https://your-org.atlassian.net"
export JIRA_TOKEN="$(cat /run/secrets/jira_token)"
export PAGERDUTY_ROUTING_KEY="$(cat /run/secrets/pd_routing_key)"

Step 1 — Ingestion & Scope Validation

The first failure mode in any scoring pipeline is dirty input: staging URLs, session-parameterised duplicates, and disallowed paths that slip through inflate error counts and distort severity tiers. Before any metric is read, enforce a clean URL manifest. Review the canonical scope parameters to confirm which environments belong in scope, and establish your crawl depth boundaries before running the validator below.

#!/usr/bin/env python3
# pipeline/01_ingest_validate.py
import os, requests, urllib.parse, json
from xml.etree import ElementTree

BASE_URL    = os.environ["SCORING_BASE_URL"]
SITEMAP_URL = os.environ["SCORING_SITEMAP_URL"]
STRIP_PARAMS = {"utm_source", "utm_medium", "utm_campaign", "sid", "session_id", "ref"}

def load_disallowed(base_url: str) -> list[str]:
    r = requests.get(f"{base_url}/robots.txt", timeout=10)
    r.raise_for_status()
    return [
        line.split(": ", 1)[1].strip()
        for line in r.text.splitlines()
        if line.startswith("Disallow:") and len(line.split(": ", 1)) > 1
    ]

def clean_url(raw: str) -> str:
    p = urllib.parse.urlparse(raw.strip())
    qs = {k: v for k, v in urllib.parse.parse_qs(p.query).items() if k not in STRIP_PARAMS}
    return urllib.parse.urlunparse((p.scheme, p.netloc, p.path, p.params,
                                    urllib.parse.urlencode(qs, doseq=True), ""))

def build_manifest(base_url: str, sitemap_url: str) -> dict:
    disallowed = load_disallowed(base_url)
    r = requests.get(sitemap_url, timeout=15)
    r.raise_for_status()
    tree = ElementTree.fromstring(r.content)
    ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    targets = []
    for loc in tree.findall(".//s:loc", ns):
        url = clean_url(loc.text)
        path = urllib.parse.urlparse(url).path
        if not any(path.startswith(d) for d in disallowed if d and d != "/"):
            targets.append({"url": url, "status": "valid", "depth": path.count("/") - 1})
    return {"manifest_version": "1.0", "url_count": len(targets), "targets": targets}

if __name__ == "__main__":
    manifest = build_manifest(BASE_URL, SITEMAP_URL)
    with open("/tmp/url_manifest.json", "w") as f:
        json.dump(manifest, f, indent=2)
    print(f"Manifest written: {manifest['url_count']} URLs")

Common mistakes:

  • Including canonicalized duplicates in the raw scoring pool — always deduplicate on the clean URL, not the raw one.
  • Ignoring HTTP 4xx/5xx rate during ingestion; a high error rate signals a crawler configuration issue, not a site health issue.
  • Failing to strip session IDs before writing the manifest — they re-introduce duplicates downstream.

Step 2 — Metric Normalisation & Baseline Calibration

Raw Core Web Vitals and crawlability metrics are dimensionally incompatible: LCP is measured in milliseconds while crawl waste is measured in URL counts. Before assigning weights you must normalise everything onto a common [0, 100] scale. Use a rolling 30-day baseline so seasonal variation does not trigger false alerts; see establishing baseline health metrics for guidance on initialising windows for recently migrated properties.

#!/usr/bin/env python3
# pipeline/02_normalise.py
import pandas as pd
import numpy as np

WINDOW_DAYS = 30
CAP_PERCENTILE = 95

def normalise_metrics(df: pd.DataFrame) -> pd.DataFrame:
    """
    Expects columns: url, raw_metric, metric_type, date.
    Returns: url, metric_type, normalised_score, percentile_rank.
    """
    df = df.sort_values("date")

    # Rolling baseline — prevents seasonal spikes from inflating scores
    df["rolling_mean"] = (
        df.groupby("metric_type")["raw_metric"]
        .transform(lambda s: s.rolling(WINDOW_DAYS, min_periods=7).mean())
    )
    df["rolling_std"] = (
        df.groupby("metric_type")["raw_metric"]
        .transform(lambda s: s.rolling(WINDOW_DAYS, min_periods=7).std().fillna(1))
    )

    # Z-score, then cap outliers at 95th percentile before scaling
    df["z_score"] = (df["raw_metric"] - df["rolling_mean"]) / df["rolling_std"]
    cap = df.groupby("metric_type")["z_score"].transform(
        lambda s: np.percentile(s.dropna(), CAP_PERCENTILE) if len(s.dropna()) else 1
    )
    df["capped"] = np.clip(df["z_score"], a_min=None, a_max=cap)

    # Min-Max to [0, 100]
    def minmax(s: pd.Series) -> pd.Series:
        lo, hi = s.min(), s.max()
        return ((s - lo) / (hi - lo + 1e-9)) * 100

    df["normalised_score"] = df.groupby("metric_type")["capped"].transform(minmax)
    df["percentile_rank"]  = df.groupby("metric_type")["normalised_score"].transform(
        lambda s: s.rank(pct=True) * 100
    )
    return df[["url", "metric_type", "normalised_score", "percentile_rank"]]

Key parameters for the normalisation step:

Parameter Type Default Purpose
WINDOW_DAYS int 30 Rolling window for baseline mean/std computation
min_periods int 7 Minimum observations before a rolling value is emitted
CAP_PERCENTILE int 95 Percentile above which z-scores are capped to suppress outliers
1e-9 epsilon float 1e-9 Prevents divide-by-zero when min equals max in a metric group

Common mistakes:

  • Using static thresholds rather than dynamic percentiles — a 3-second LCP is fine for a B2B tool but catastrophic on a news article.
  • Applying uniform scaling across LCP, CLS, and INP together; normalise each metric type independently within its own distribution.
  • Ignoring the min_periods guard — early in a new domain's life, rolling windows with fewer than 7 observations produce extreme z-scores that inflate severity.

Step 3 — Risk Matrix & Weight Assignment

Weights must reflect revenue impact, not engineering preference. LCP and INP regressions on conversion-path pages carry far higher business cost than the same regression on a blog archive. Pair each metric weight with a page-level impact multiplier derived from your crawl depth and scope configuration, where shallow revenue-critical URLs receive the highest multipliers.

Severity against frequency A two by two grid. The vertical axis is how severe a single instance is; the horizontal is how often it occurs, measured as request share rather than raw count so a low-traffic site does not over-promote sparse errors. Severe and frequent is P0. Severe and rare is P1, investigated on its own merits. Mild and frequent is P2, usually a template fix. Mild and rare is tracked but not scheduled. severe mild P1 — investigate rare but serious: a payment path 5xx that only fires under one condition P0 — page now serious and constant: this is what the whole framework exists to find P3 — track only mild and rare: record it in the trend series and move on P2 — template fix mild but everywhere: one change closes the whole class at once rare frequent Frequency is request share, not raw count. A raw count promotes whatever the busiest section happens to emit, which on most sites is the section that already gets attention. 
# config/risk_matrix.yaml
version: "2.1"
weights:
  lcp:               0.25   # Largest Contentful Paint — direct ranking signal
  cls:               0.20   # Cumulative Layout Shift — UX & conversion impact
  inp:               0.20   # Interaction to Next Paint — engagement signal
  wcag_violations:   0.15   # Accessibility issues — legal risk + indexability
  crawl_budget_waste: 0.20  # Orphaned / low-value pages consuming crawl quota

impact_multipliers:
  revenue_critical:  2.5    # Checkout, pricing, product detail pages
  conversion_path:   1.8    # Lead-gen, sign-up, key landing pages
  informational:     1.0    # Blog posts, FAQs, docs
  orphaned:          0.6    # Pages with no inbound internal links

severity_tiers:
  critical: 85   # Triggers immediate Jira + PagerDuty
  high:     65   # Tickets to current sprint backlog
  medium:   40   # Logged to weekly review queue
  low:       0   # Monitored; no immediate action
#!/usr/bin/env python3
# pipeline/03_score.py
"""
Composite Risk Score = Σ(weight_i × normalised_metric_i) × impact_multiplier
"""
import yaml, pandas as pd
from pathlib import Path

CONFIG = yaml.safe_load(Path("/abs/config/risk_matrix.yaml").read_text())
WEIGHTS = CONFIG["weights"]
MULTIPLIERS = CONFIG["impact_multipliers"]
TIERS = CONFIG["severity_tiers"]

def assign_tier(score: float) -> str:
    if score >= TIERS["critical"]: return "critical"
    if score >= TIERS["high"]:     return "high"
    if score >= TIERS["medium"]:   return "medium"
    return "low"

def compute_risk_scores(normalised: pd.DataFrame, page_meta: pd.DataFrame) -> pd.DataFrame:
    """
    normalised:  [url, metric_type, normalised_score]
    page_meta:   [url, page_class]  — page_class maps to impact_multipliers keys
    """
    pivot = normalised.pivot_table(index="url", columns="metric_type",
                                   values="normalised_score", aggfunc="mean").fillna(0)
    for col in WEIGHTS:
        if col not in pivot.columns:
            pivot[col] = 0.0

    pivot["weighted_sum"] = sum(pivot[m] * w for m, w in WEIGHTS.items())

    result = pivot[["weighted_sum"]].join(page_meta.set_index("url"), how="left")
    result["impact_multiplier"] = result["page_class"].map(MULTIPLIERS).fillna(1.0)
    result["risk_score"] = (result["weighted_sum"] * result["impact_multiplier"]).clip(0, 100)
    result["severity_tier"] = result["risk_score"].apply(assign_tier)
    return result.reset_index()[["url", "risk_score", "severity_tier", "impact_multiplier"]]

The BigQuery query below validates your weight choices: if the revenue_impact_delta for critical-tier pages is not at least 15–20 percentage points below baseline conversion, your weights are likely miscalibrated.

-- BigQuery: verify weight calibration via historical conversion correlation
SELECT
    metric_type,
    ROUND(AVG(conversion_rate) * 100, 2)                                              AS baseline_conv_pct,
    ROUND(AVG(CASE WHEN risk_score >= 85 THEN conversion_rate END) * 100, 2)          AS critical_conv_pct,
    ROUND(
        SAFE_DIVIDE(
            AVG(conversion_rate) - AVG(CASE WHEN risk_score >= 85 THEN conversion_rate END),
            AVG(conversion_rate)
        ) * 100, 2
    )                                                                                  AS revenue_delta_pct
FROM `${BQ_PROJECT}.${BQ_DATASET}.audit_metrics`
WHERE date BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY) AND CURRENT_DATE()
GROUP BY metric_type
ORDER BY revenue_delta_pct DESC

Step 4 — Execution, Scheduling & Alert Routing

Scores are only useful if they are acted upon automatically. Run an incremental daily job plus a full weekly recalculation; route critical issues to Jira immediately and trigger PagerDuty when the score velocity (rate of change) exceeds a threshold — a sudden 20-point jump indicates a deployment regression rather than a gradual drift.

# .github/workflows/risk-scoring-pipeline.yml
name: Daily Risk Recalculation & Routing
on:
  schedule:
    - cron: '0 2 * * *'   # 02:00 UTC daily
  workflow_dispatch:

env:
  BQ_PROJECT:    ${{ secrets.BQ_PROJECT }}
  BQ_DATASET:    audit_metrics
  JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
  JIRA_TOKEN:    ${{ secrets.JIRA_TOKEN }}
  PD_ROUTING_KEY: ${{ secrets.PD_ROUTING_KEY }}

jobs:
  score-and-route:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -r requirements.txt

      - name: Build URL manifest
        run: |
          set -euo pipefail
          python pipeline/01_ingest_validate.py

      - name: Normalise and score
        run: |
          set -euo pipefail
          python pipeline/02_normalise.py --input /tmp/url_manifest.json
          python pipeline/03_score.py --output /tmp/risk_manifest.json

      - name: Route critical issues to Jira
        run: |
          set -euo pipefail
          jq -r '.items[] | select(.severity_tier == "critical") | .url' /tmp/risk_manifest.json \
          | while IFS= read -r url; do
              curl -sf -X POST "${JIRA_BASE_URL}/rest/api/3/issue" \
                -H "Authorization: Bearer ${JIRA_TOKEN}" \
                -H "Content-Type: application/json" \
                -d "{\"fields\":{\"project\":{\"key\":\"TECH\"},\"summary\":\"Critical Risk: ${url}\",\"issuetype\":{\"name\":\"Bug\"},\"priority\":{\"name\":\"Highest\"}}}"
            done

      - name: Alert PagerDuty on score-velocity spike
        run: |
          set -euo pipefail
          # Fire alert if any URL's score increased by ≥20 points vs prior run
          VELOCITY=$(jq '[.items[] | select(.score_delta >= 20)] | length' /tmp/risk_manifest.json)
          if [ "${VELOCITY}" -gt 0 ]; then
            curl -sf -X POST "https://events.pagerduty.com/v2/enqueue" \
              -H "Content-Type: application/json" \
              -d "{\"routing_key\":\"${PD_ROUTING_KEY}\",\"event_action\":\"trigger\",\"payload\":{\"summary\":\"${VELOCITY} URLs show ≥20pt risk-score spike\",\"severity\":\"critical\",\"source\":\"risk-scoring-pipeline\"}}"
          fi

      - name: Upload risk manifest artefact
        uses: actions/upload-artifact@v4
        with:
          name: risk-manifest-${{ github.run_id }}
          path: /tmp/risk_manifest.json
          retention-days: 90

Concurrency guard — prevent parallel runs from overwriting the same manifest:

concurrency:
  group: risk-scoring-pipeline
  cancel-in-progress: false   # queue rather than cancel; never discard a run

Step 5 — Artefact Capture & Storage

The risk manifest and supporting Parquet files form an audit trail for trend analysis and tracking metric trends across release cycles. Without versioned artefacts you cannot correlate a score spike to a specific deployment SHA.

#!/usr/bin/env python3
# pipeline/05_persist_artefacts.py
"""
Write risk_manifest.json and risk_scores.parquet to a versioned GCS path.
Path convention: gs://<bucket>/risk-scores/<YYYY-MM-DD>/<git_sha>/
"""
import os, json, datetime, subprocess
from pathlib import Path
import pandas as pd
from google.cloud import storage

BUCKET      = os.environ["GCS_BUCKET"]         # e.g. "my-audit-artefacts"
MANIFEST_IN = Path("/tmp/risk_manifest.json")

def get_git_sha() -> str:
    return subprocess.check_output(["git", "rev-parse", "--short", "HEAD"],
                                   text=True).strip()

def upload(local_path: Path, gcs_key: str) -> None:
    client = storage.Client()
    bucket = client.bucket(BUCKET)
    blob   = bucket.blob(gcs_key)
    blob.upload_from_filename(str(local_path))
    print(f"Uploaded: gs://{BUCKET}/{gcs_key}")

if __name__ == "__main__":
    today  = datetime.date.today().isoformat()
    sha    = get_git_sha()
    prefix = f"risk-scores/{today}/{sha}"

    # JSON manifest
    upload(MANIFEST_IN, f"{prefix}/risk_manifest.json")

    # Parquet for BI / BigQuery external table
    manifest = json.loads(MANIFEST_IN.read_text())
    df = pd.DataFrame(manifest["items"])
    parquet_path = Path("/tmp/risk_scores.parquet")
    df.to_parquet(parquet_path, index=False, compression="snappy")
    upload(parquet_path, f"{prefix}/risk_scores.parquet")

Retention policy: keep daily manifests for 90 days; weekly full-recalculation exports indefinitely. Set this on the GCS bucket lifecycle rule, not in code.

Verification Checklist

Run these checks after each pipeline execution to confirm the workflow produced valid output:

  1. Manifest URL count matches expectation — compare url_count in /tmp/url_manifest.json against the previous run's count. A drop greater than 5% indicates a sitemap regression.
  2. Score distribution sanity — fewer than 2% of URLs should be critical tier on a stable site; if the percentage is higher, inspect the weight config for miscalibration.
  3. Jira deduplication — query Jira for open TECH issues with summaries matching Critical Risk: and confirm no URL received two tickets in the same run.
  4. Artefact checksum — verify the GCS object's md5Hash matches a local md5sum /tmp/risk_manifest.json before closing the pipeline run.
  5. BigQuery row count — confirm the row count in audit_metrics increased by the expected URL count:
    SELECT COUNT(*) FROM `${BQ_PROJECT}.${BQ_DATASET}.audit_metrics`
WHERE date = CURRENT_DATE()
  6. PagerDuty alert closed — if a velocity spike alert fired, confirm it resolves automatically when the next run produces no spikes (event_action: resolve).

Troubleshooting

Score distribution is heavily skewed toward critical Root cause: weight config and/or impact multipliers applied to a URL set that includes staging or parameterised pages that escaped the manifest filter. Fix:

jq '[.targets[] | select(.url | test("staging|dev|\\?.*sid="))] | length' /tmp/url_manifest.json
# If non-zero, re-run 01_ingest_validate.py with an extended STRIP_PARAMS list
Three ways the inputs lie Three rows. A CDN edge cache masking origin 5xx errors means the access log shows no server errors while users see them, which is fixed by logging the upstream status. A missing PyYAML dependency stops the routing manifest being written at all, which is a hard failure rather than a silent one. A headless browser that cannot launch in CI means the JavaScript-rendered half of the estate is never scored, leaving a partial risk picture that looks complete. SYMPTOM ROOT CAUSE FIX No 5xx in the logs while users report intermittent server errors The edge cache masks them the CDN served a cached page over a failing origin response Log upstream_status add it to log_format and reload, then flush the affected paths The manifest is never written and the routing step has nothing to consume A missing dependency PyYAML is not installed in the environment the step runs in Pin it in the lockfile so the environment is built the same way everywhere Half the estate is unscored and the risk picture looks cleaner than it is The browser will not launch missing system libraries or a sandbox the container disallows Install and pin system deps plus a pinned browser binary in the CI image All three produce a risk report with no obvious gap in it. The first is the worst: it reports zero server errors on a site that is actively failing for users.

Rolling baseline returns NaN for new domains Root cause: fewer than min_periods observations in the rolling window on day 1–6 of monitoring. Fix: seed the baseline using historical Lighthouse CI data or set min_periods=1 for the first two weeks, then raise it to 7.

# Temporary override in pipeline/02_normalise.py
WINDOW_MIN_PERIODS = int(os.getenv("BASELINE_MIN_PERIODS", "7"))

Jira tickets created for already-resolved URLs Root cause: deduplication logic is missing — the pipeline checks current risk_score but not whether an open Jira issue already exists for the URL. Fix: query the Jira search API before creating each ticket:

curl -sf "${JIRA_BASE_URL}/rest/api/3/search?jql=project=TECH+AND+summary~\"${ENCODED_URL}\"+AND+statusCategory!=Done" \
  -H "Authorization: Bearer ${JIRA_TOKEN}" | jq '.total'
# Only POST if total == 0

PagerDuty alert never resolves after a fix Root cause: the pipeline only fires trigger events, never resolve. Without a matching resolve event, PagerDuty holds the incident open indefinitely. Fix: add a resolve step that fires when VELOCITY == 0 and the previous run had a spike:

# Store run velocity in GCS as a state file; compare to prior run
PREV_VELOCITY=$(gsutil cat "gs://${GCS_BUCKET}/state/last_velocity.txt" 2>/dev/null || echo "0")
if [ "${PREV_VELOCITY}" -gt 0 ] && [ "${VELOCITY}" -eq 0 ]; then
  curl -sf -X POST "https://events.pagerduty.com/v2/enqueue" \
    -H "Content-Type: application/json" \
    -d "{\"routing_key\":\"${PD_ROUTING_KEY}\",\"event_action\":\"resolve\",\"dedup_key\":\"risk-velocity-spike\"}"
fi
echo "${VELOCITY}" | gsutil cp - "gs://${GCS_BUCKET}/state/last_velocity.txt"

Parquet export fails with ArrowInvalid on mixed-type columns Root cause: normalised_score column contains a mix of float and None when some URLs have no metrics for a given type. Fix:

df["normalised_score"] = pd.to_numeric(df["normalised_score"], errors="coerce").astype("float32")

BigQuery load job times out on large manifests Root cause: writing row-by-row via the streaming API instead of bulk load from GCS Parquet. Fix: always write the Parquet file first and load from GCS:

bq load --source_format=PARQUET --replace \
  "${BQ_PROJECT}:${BQ_DATASET}.audit_metrics" \
  "gs://${GCS_BUCKET}/risk-scores/$(date +%F)/*/risk_scores.parquet"

FAQ

How often should the risk score be recalculated?

Run an incremental delta pass daily (processing only URLs touched by recent crawl diffs) and a full recalculation weekly. Trigger on-demand runs after any significant deployment, migration, or sitemap structural change. The quarterly audit schedule describes how to align scoring cadence with wider audit cycles.

Can I apply the same weight config to a blog and an e-commerce catalogue?

No. Blogs prioritise crawl coverage and LCP; catalogues weight CLS, INP, and structured-data validity more heavily. Maintain separate risk_matrix.yaml files per site section and select the correct config at scoring time based on the URL's page_class field. See calibrating error thresholds for different site sections for a worked example.

What is the difference between a critical and a high severity tier?

critical (score ≥ 85) means immediate revenue or crawlability impact — open an incident and halt deployments to affected templates. high (65–84) means a degradation that will surface in search performance within days; fix within the next sprint without a full incident declaration. Triage decisions for these tiers are expanded in prioritizing critical vs non-critical site errors.

How do I prevent score inflation from seasonal traffic spikes?

Use a 30-day rolling baseline with min_periods=7 as the default. For sites with strong year-over-year seasonality — e-commerce during peak, news during elections — supplement the rolling window with a year-over-year delta comparison. Query the same date range from the prior year in BigQuery and blend the baselines with a 0.7 / 0.3 weighting in favour of the recent window.