17 min read

Metric Scoring & Data Normalization

Automated site health monitoring fails not from a lack of data but from a lack of determinism. Raw telemetry from crawlers, synthetic tests, and real-user monitoring arrives in incompatible units, sampled at different device tiers, and timestamped across misaligned timezone boundaries. This reference covers the full pipeline that SREs, SEO engineers, and agency teams use to transform that raw signal into reproducible composite health scores, actionable threshold alerts, and deployment-gating CI checks.

Pipeline Architecture: Ingest → Normalize → Score → Alert → Remediate

The diagram below shows the ordered execution chain covered by this section. Each stage has a corresponding guide that covers its configuration in detail.

Metric scoring pipeline stages Sequential flow diagram showing five pipeline stages: Ingest raw telemetry, Normalize cross-device data, Score weighted metrics, Alert on threshold breaches, and Remediate via tickets and CI gates. Ingest Crawlers · RUM · Logs Normalize Device · Units · URLs Score Weighted composite Alert Threshold · Anomaly Remediate Tickets · CI gates
Stage Component Function
Ingest Crawler / Log ingestion Extracts DOM snapshots, network waterfalls, JS traces
Normalize Cross-device calibration engine Aligns LCP, CLS, INP distributions across mobile and desktop
Score Weighted metric aggregation Computes composite indices from normalised telemetry
Alert Threshold breach router Triggers PagerDuty / Slack on SLO violations
Remediate Automated ticket & CI gate Opens Jira / Linear tickets and blocks deployments

Phase 1 — Raw Data Ingestion and Pre-Processing

Problem framing

Every downstream scoring decision is only as reliable as the data entering the pipeline. Ingesting unfiltered scanner traffic, misresolved redirect targets, or duplicate session URLs corrupts baseline percentiles immediately. The ingestion layer must enforce a canonical URL contract before any metric value is recorded.

Route headless crawlers, server access logs, RUM APIs, and Lighthouse CI runners into a single queue. The automated crawling pipeline provides the upstream tooling that feeds this stage; establish strict extraction schemas before connecting any new source.

#!/usr/bin/env python3
"""
ingest/validate.py — canonical URL normalisation and payload validation.
Requires: jsonschema>=4.21.0  (pin in requirements.txt)
"""
import re
import json
import jsonschema
from pathlib import Path

# Strip UTM parameters, session IDs, and click-tracking suffixes
PARAM_PATTERN = re.compile(
    r'[?&](utm_\w+|sid|session_id|fbclid|gclid|mc_\w+)=[^&]*',
    re.IGNORECASE
)

# CLS is unitless with no hard ceiling; practical upper bound set at 2.0
CRAWL_SCHEMA = {
    "type": "object",
    "required": ["url", "status_code", "lcp_ms", "cls_score", "inp_ms"],
    "properties": {
        "url":         {"type": "string", "format": "uri"},
        "status_code": {"type": "integer", "minimum": 200, "maximum": 599},
        "lcp_ms":      {"type": "number",  "minimum": 0},
        "cls_score":   {"type": "number",  "minimum": 0, "maximum": 2.0},
        "inp_ms":      {"type": "number",  "minimum": 0}
    },
    "additionalProperties": True
}

def normalize_url(raw_url: str) -> str:
    """Strip tracking params and trailing slashes for canonical key derivation."""
    clean = PARAM_PATTERN.sub('', raw_url)
    clean = re.sub(r'[?&]$', '', clean)      # remove dangling separator
    return clean.rstrip('/')

def validate_payload(record: dict) -> bool:
    try:
        jsonschema.validate(instance=record, schema=CRAWL_SCHEMA)
        return True
    except jsonschema.ValidationError as exc:
        print(f"[WARN] Payload rejected — {exc.message}: {json.dumps(record)[:200]}")
        return False

def ingest_batch(records: list[dict]) -> list[dict]:
    clean = []
    for r in records:
        r["url"] = normalize_url(r["url"])
        if validate_payload(r):
            clean.append(r)
    return clean

Verification steps:

  1. Run python -m pytest tests/test_ingest.py -v — all URL normalisation cases must pass.
  2. Check queue depth in your ingestion broker; it should drain within the crawl cadence window.
  3. Spot-check five random records from the normalised stream against source URLs to confirm canonical resolution.

Common mistakes:

  • Ingesting unfiltered bot and scanner traffic into baseline calculations — segment by user_agent before the validation gate.
  • Failing to resolve relative URLs before normalisation — convert to absolute paths using the crawl origin as base.
  • Overlooking timezone discrepancies in server log timestamps — parse all timestamps to UTC at ingestion, not at query time.

Phase 2 — Cross-Device Normalisation and Baseline Calibration

Deterministic device profiles

Mobile and desktop LCP distributions are not comparable without a normalisation step. A 3 000 ms LCP on a throttled mobile profile is not equivalent to 3 000 ms on a fibre-connected desktop. For normalizing performance data across device types, map every synthetic run to a named profile with explicit CPU throttle multipliers and network constraints.

# lighthouserc.yml — pin lighthouse version in package.json (e.g. "lighthouse": "12.3.0")
ci:
  collect:
    numberOfRuns: 5           # median of 5 reduces single-run variance
    settings:
      chromeFlags: "--no-sandbox --disable-dev-shm-usage"
  profiles:
    mobile:
      preset: "mobile"
      throttling:
        cpuSlowdownMultiplier: 4
        rttMs: 150
        throughputKbps: 1638   # Equivalent to slow 4G
    desktop:
      preset: "desktop"
      throttling:
        cpuSlowdownMultiplier: 1
        rttMs: 40
        throughputKbps: 10240

Rolling baseline computation

Compute device-class baselines using a 30-day rolling window so seasonal traffic patterns do not corrupt the reference distribution. The SQL below targets BigQuery but the PERCENTILE_CONT window function is standard SQL and portable to PostgreSQL or DuckDB.

-- Rolling p75 baseline per device class, updated daily
-- Table: `project.dataset.raw_telemetry`
SELECT
    device_type,
    metric_name,
    DATE(collection_ts)                                                 AS collection_date,
    PERCENTILE_CONT(metric_value, 0.75) OVER (
        PARTITION BY device_type, metric_name
        ORDER BY UNIX_DATE(DATE(collection_ts))
        RANGE BETWEEN 29 PRECEDING AND CURRENT ROW
    )                                                                   AS p75_baseline,
    PERCENTILE_CONT(metric_value, 0.90) OVER (
        PARTITION BY device_type, metric_name
        ORDER BY UNIX_DATE(DATE(collection_ts))
        RANGE BETWEEN 29 PRECEDING AND CURRENT ROW
    )                                                                   AS p90_baseline
FROM `project.dataset.raw_telemetry`
WHERE collection_ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 60 DAY)

Idempotency guard: Store the computed baseline snapshot keyed on (device_type, metric_name, snapshot_date) with a UNIQUE constraint. Re-running the pipeline on the same day overwrites the same row rather than appending a duplicate.

Common mistakes:

  • Using raw desktop metrics as universal baselines for mobile-first indexing — always segment before computing percentiles.
  • Ignoring network latency variance in synthetic testing environments — use a fixed emulation profile, not the host machine's real connection.
  • Hardcoding device profiles inline instead of parameterising via environment variables — treat CPU_THROTTLE_MULTIPLIER and NETWORK_PROFILE as config, not constants.

Phase 3 — Threshold Configuration and Algorithmic Scoring

Section-aware thresholds

Uniform thresholds fail because checkout flows, authentication pages, and blog content have different conversion stakes. Calibrating thresholds for different site sections covers the full workflow; the JSON below shows the minimum viable threshold matrix for two section types.

{
  "version": "2.1.0",
  "thresholds": {
    "checkout": {
      "lcp_ms":   {"info": 2000, "warning": 2500, "critical": 4000},
      "inp_ms":   {"info": 150,  "warning": 200,  "critical": 500},
      "cls_score":{"info": 0.05, "warning": 0.1,  "critical": 0.25}
    },
    "content": {
      "lcp_ms":   {"info": 2500, "warning": 3000, "critical": 5000},
      "inp_ms":   {"info": 200,  "warning": 300,  "critical": 600},
      "cls_score":{"info": 0.1,  "warning": 0.2,  "critical": 0.4}
    },
    "auth": {
      "lcp_ms":   {"info": 2000, "warning": 2500, "critical": 3500},
      "inp_ms":   {"info": 100,  "warning": 200,  "critical": 400},
      "cls_score":{"info": 0.05, "warning": 0.1,  "critical": 0.2}
    }
  }
}

Store this file in version control alongside your application code. Any PR that modifies threshold values should trigger a scoring-engine diff review.

Weighted composite scoring

The custom health score algorithm design workflow explains how to derive per-section weight matrices. The function below implements the composite scorer with a regression penalty.

#!/usr/bin/env python3
"""
scoring/composite.py — weighted health score with regression decay.
"""
from __future__ import annotations
from typing import Any

# Default weights: adjust per section via scored_sections config
DEFAULT_WEIGHTS: dict[str, float] = {
    "lcp_score":   0.40,
    "inp_score":   0.35,
    "cls_score":   0.15,
    "ttfb_score":  0.10,
}

def normalise_metric(value: float, warning: float, critical: float) -> float:
    """Map raw metric value to [0, 100] score; higher = better."""
    if value <= warning:
        # Linear scale from 100 down to 50 between 0 and warning
        return 100.0 - (value / warning) * 50.0
    if value <= critical:
        # Linear scale from 50 down to 0 between warning and critical
        return 50.0 - ((value - warning) / (critical - warning)) * 50.0
    return 0.0

def calculate_weighted_score(
    raw_metrics: dict[str, float],
    thresholds: dict[str, dict[str, float]],
    weights: dict[str, float] | None = None,
    regression_count: int = 0,
    decay: float = 0.95,
) -> float:
    """Return a 0–100 composite health score with optional regression penalty."""
    w = weights or DEFAULT_WEIGHTS
    normalised: dict[str, float] = {}

    metric_map = {
        "lcp_score":  ("lcp_ms",   "warning", "critical"),
        "inp_score":  ("inp_ms",   "warning", "critical"),
        "cls_score":  ("cls_score","warning", "critical"),
        "ttfb_score": ("ttfb_ms",  "warning", "critical"),
    }
    for score_key, (raw_key, warn_key, crit_key) in metric_map.items():
        if raw_key in raw_metrics and score_key in thresholds:
            t = thresholds[score_key]
            normalised[score_key] = normalise_metric(
                raw_metrics[raw_key], t[warn_key], t[crit_key]
            )

    raw_score = sum(normalised.get(k, 0) * v for k, v in w.items())
    penalty = decay ** regression_count if regression_count > 0 else 1.0
    return round(max(0.0, min(100.0, raw_score * penalty)), 2)

Verification steps:

  1. Unit-test normalise_metric(2500, 2500, 4000) — should return exactly 50.0.
  2. Confirm the composite score for a perfect payload is 100.0 (all metrics at zero).
  3. Verify that regression_count=5 with decay=0.95 yields approximately 77 % of the base score.

Common mistakes:

  • Applying uniform thresholds across pages with fundamentally different user journeys.
  • Over-weighting vanity metrics (e.g. raw byte size) over conversion-critical signals (INP on checkout).
  • Failing to version-control threshold JSON alongside application code — threshold drift causes silent scoring regressions.

Phase 4 — CI/CD Integration and Release Cycle Tracking

Embedding audit runners into pull-request pipelines

Integrate the scoring engine into your CI workflow on pull-request merges, not on every commit — full audits are expensive and noise increases false-positive block rates. Tracking metric trends across release cycles covers the correlation of deployment tags with score deltas; the workflow below wires the audit run to the CI merge event.

# .github/workflows/site-health-audit.yml
name: Site Health Audit
on:
  pull_request:
    branches: [main, "release/**"]
    paths:
      - "src/**"
      - "public/**"
      - "package*.json"
  workflow_dispatch:          # allow manual runs for hotfix validation

env:
  NODE_VERSION: "20.x"
  LIGHTHOUSE_VERSION: "12.3.0"
  AUDIT_BASELINE_BUCKET: "gs://your-bucket/baselines"

jobs:
  audit:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Warm cache (eliminate cold-start noise)
        run: |
          TARGET_URL="${{ vars.STAGING_URL }}"
          for i in 1 2 3; do curl -s -o /dev/null "$TARGET_URL"; done

      - name: Run Lighthouse CI
        run: |
          npx --yes "lighthouse@${{ env.LIGHTHOUSE_VERSION }}" \
            "${{ vars.STAGING_URL }}" \
            --output=json \
            --output-path=./audit-results/lhci-report.json \
            --chrome-flags="--no-sandbox --disable-dev-shm-usage"

      - name: Diff against baseline
        run: node scripts/diff-metrics.js \
          --report ./audit-results/lhci-report.json \
          --baseline "${{ env.AUDIT_BASELINE_BUCKET }}/main-latest.json" \
          --threshold-config config/scoring/thresholds.json
        env:
          GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GCP_SA_KEY }}

      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: audit-results-${{ github.sha }}
          path: ./audit-results/
          retention-days: 90

Webhook payload for alert routing

When a score crosses a critical threshold, the scoring engine emits a structured webhook. Include a stable dedup_key (hash of metric name + deployment tag) so downstream PagerDuty or Opsgenie can suppress duplicate alerts within a single deployment window.

{
  "event": "metric_regression",
  "severity": "critical",
  "section": "checkout",
  "metrics": {
    "lcp_ms": {"value": 4200, "threshold_critical": 4000, "delta_pct": 5.0},
    "inp_ms": {"value": 550,  "threshold_critical": 500,  "delta_pct": 10.0}
  },
  "deployment_tag": "v2.4.1-rc3",
  "baseline_snapshot": "2026-06-20",
  "dedup_key": "sha256:8f3ac14...c91b"
}

Rate-limiting guard: Wrap the alert emitter in a per-dedup_key TTL cache (e.g. Redis SET NX EX 3600) to prevent alert floods during a deployment that triggers multiple consecutive threshold breaches.

Common mistakes:

  • Running full multi-URL audits on every commit — scope the trigger to PR merges and include a paths: filter.
  • Failing to isolate STAGING_URL and PRODUCTION_URL environment variables — a mis-pointed audit pollutes production baselines.
  • Skipping the cache warm-up requests before Lighthouse runs — cold-start TTFB readings inflate LCP and skew composite scores.

Phase 5 — Validation, Forecasting, and Automated Remediation

Scoring accuracy validation against RUM

Synthetic audit scores must be validated against real-user monitoring telemetry before being treated as authoritative. Correlate p75 LCP from the Chrome UX Report (CrUX) or your own RUM SDK against your synthetic composite score on a weekly cadence. A synthetic score that consistently diverges from RUM by more than 15 percentage points signals a miscalibrated device profile or environment.

Time-series forecasting for proactive breach detection

Use a time-series model to predict which pages are trending toward a critical threshold before the next deployment. The Prophet configuration below is suitable for weekly-seasonal traffic patterns.

#!/usr/bin/env python3
"""
forecasting/lcp_breach.py — LCP threshold breach forecast.
Requires: prophet>=1.1.5, pandas>=2.0.0  (pin in requirements.txt)
"""
from __future__ import annotations
import pandas as pd
from prophet import Prophet   # type: ignore[import]

CRITICAL_LCP_MS = 4000.0

def forecast_lcp_breach(
    historical_df: pd.DataFrame,   # columns: date (datetime), lcp_ms (float)
    horizon_days: int = 14,
) -> pd.DataFrame:
    """
    Returns forecast DataFrame with columns:
    ds, yhat, yhat_lower, yhat_upper, will_breach (bool)
    """
    model = Prophet(
        yearly_seasonality=False,
        weekly_seasonality=True,
        daily_seasonality=False,
        changepoint_prior_scale=0.05,    # conservative — site metrics are slow-moving
        interval_width=0.90,
    )
    fit_df = historical_df.rename(columns={"date": "ds", "lcp_ms": "y"})
    model.fit(fit_df)
    future = model.make_future_dataframe(periods=horizon_days)
    forecast = model.predict(future)
    result = forecast[["ds", "yhat", "yhat_lower", "yhat_upper"]].copy()
    result["will_breach"] = result["yhat_upper"] >= CRITICAL_LCP_MS
    return result

Automated remediation ticket creation

When a threshold breach is confirmed (not just forecast), open a structured remediation ticket. Gate automated ticket creation behind a confidence check — only fire for breaches that persist across two consecutive audit runs to eliminate transient spikes.

#!/usr/bin/env python3
"""
remediation/jira_ticket.py — automated remediation ticket via Jira REST API v3.
"""
from __future__ import annotations
import os
import requests

JIRA_BASE_URL = os.environ["JIRA_BASE_URL"]        # e.g. https://your-org.atlassian.net
JIRA_TOKEN    = os.environ["JIRA_API_TOKEN"]
JIRA_PROJECT  = os.environ.get("JIRA_PROJECT_KEY", "SITE-OPS")

def create_remediation_ticket(
    metric: str,
    value: float,
    threshold: float,
    deployment_tag: str,
    section: str,
) -> str:
    """Opens a Jira Bug ticket; returns the created issue key (e.g. SITE-OPS-42)."""
    summary = (
        f"[Auto] {metric} breach on {section}: "
        f"{value:.0f} > {threshold:.0f} @ {deployment_tag}"
    )
    payload = {
        "fields": {
            "project":     {"key": JIRA_PROJECT},
            "summary":     summary,
            "description": {
                "type": "doc", "version": 1,
                "content": [{"type": "paragraph", "content": [
                    {"type": "text", "text": (
                        f"Automated detection: {metric} value {value:.0f} exceeds "
                        f"critical threshold {threshold:.0f} on deployment {deployment_tag}. "
                        "Review diff-metrics report attached to the CI run."
                    )}
                ]}]
            },
            "issuetype":   {"name": "Bug"},
            "priority":    {"name": "High"},
            "labels":      ["site-health", "automated", section],
        }
    }
    headers = {
        "Authorization": f"Bearer {JIRA_TOKEN}",
        "Content-Type":  "application/json",
    }
    resp = requests.post(
        f"{JIRA_BASE_URL}/rest/api/3/issue",
        json=payload, headers=headers, timeout=15
    )
    resp.raise_for_status()
    return resp.json()["key"]

Rollback procedure: If automated tickets are opening for a legitimate threshold change (not a real regression), set REMEDIATION_DRY_RUN=true in your CI environment to log ticket payloads without calling the Jira API, then update the threshold JSON in a follow-up PR.

Cross-Cutting Concerns

Data retention and versioning

Artifact Format Retention Storage
Raw telemetry records Parquet, gzip compressed 90 days Object storage (versioned bucket)
Computed baselines JSON, keyed by snapshot date 1 year Object storage
Composite score time-series Parquet 2 years Data warehouse
Audit run reports JSON + HTML 90 days CI artifact store
  • Pin all crawler and audit tool versions in lockfiles (package-lock.json, requirements.txt). Any version bump must be a deliberate PR, not an incidental upgrade.
  • Containerise the scoring engine with a deterministic base image (pin the digest, not just the tag).
  • Keep threshold configuration files (thresholds.json) and scoring weights under the same version-control review workflow as application code.

Environment parity

  • Enforce identical NODE_ENV, LIGHTHOUSE_VERSION, and device-profile settings across local, staging, and CI environments using a .env.audit file checked into source control (secrets injected at runtime, not committed).
  • Use docker run --rm to execute audit scripts locally and in CI from the same image — eliminates "works on my machine" scoring variance.
  • Establish a scheduled weekly audit against production (not just PR-triggered staging audits) to detect organic performance drift that deployment diffs would miss.

Failure Modes and Rollback

The following failure patterns are specific to metric scoring pipelines. Each entry includes a root cause, a diagnostic command, and a recovery path.

1. Baseline drift from cold-start noise Root cause: audit runs execute before server-side caches warm up, inflating TTFB and LCP values. Diagnostic: compare p50_lcp on the first run vs. the median of runs 2–5 in a multi-run batch. Fix: prepend three curl -s -o /dev/null "$URL" warm-up requests before each Lighthouse invocation.

2. Device-class contamination in percentile computation Root cause: mobile and desktop telemetry pooled in the same PERCENTILE_CONT window. Diagnostic: SELECT device_type, COUNT(*) FROM raw_telemetry GROUP BY device_type — any single-value result means the partition split failed. Fix: verify the PARTITION BY device_type clause is present and that the ETL job populates device_type before the baseline query runs.

3. Scoring engine version skew between CI and production Root cause: package.json specifies a range (^12) rather than an exact version; npm resolves a newer minor that changes scoring weights. Fix: npm install [email protected] --save-exact and commit the updated package-lock.json.

4. Alert flood on legitimate threshold change Root cause: threshold JSON updated without clearing the alert deduplication cache. Fix: redis-cli DEL "dedup:*" (or equivalent key namespace flush) after deploying a threshold change.

5. Jira API token expiry causing silent ticket loss Root cause: bearer token rotated without updating the CI secret. Diagnostic: curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $JIRA_API_TOKEN" "$JIRA_BASE_URL/rest/api/3/myself" — expect 200. Fix: rotate the token in Jira, update JIRA_API_TOKEN in your CI secrets store, re-run the failed pipeline.

6. Prophet forecast instability on short history Root cause: fewer than 30 data points in the training window; the model overfits changepoints. Fix: set changepoint_prior_scale=0.001 and disable weekly_seasonality until you have at least 60 daily data points; fall back to a simple linear trend in the interim.

FAQ

What is the correct percentile to use as a scoring baseline?

Use p75 for LCP and INP thresholds, matching the Chrome UX Report and Core Web Vitals programme definitions. Use p90 for server-side TTFB when your SLO budget is tighter. Always segment by device class before computing the percentile — pooling mobile and desktop inflates apparent desktop headroom and masks real mobile regressions.

How do I prevent scoring drift between staging and production?

Pin the exact audit tool version in both environments, run cache warm-up requests before each measurement window, and store scoring engine logic in version control alongside application code so any change triggers a diff alert. Schedule a weekly production audit run to catch organic drift that PR-triggered staging audits miss.

Should LCP, CLS, and INP carry equal weight in a composite health score?

No. Weight by conversion impact for the page template: checkout pages should penalise INP regressions most heavily, content pages should weight LCP first. Use the custom health score algorithm workflow to configure per-section weight matrices and document the rationale alongside the threshold JSON.