12 min read

Aggregating Scores Across URL Segments

A single site-wide health score tells you almost nothing about where to act. A composite of 82 can mean every template is a consistent 82, or it can mean a fast marketing section dragging up a checkout flow stuck at 41. This page is part of Building Score Aggregation Pipelines and covers the specific step of collapsing thousands of per-URL scores into path-based segment scores — one number per /blog, /product, or /checkout — without letting sparsely measured segments masquerade as healthy ones.

The scores being rolled up here come from the per-URL composite described in Designing Custom Health Score Algorithms; this page assumes that work is already done and focuses only on the segment-level aggregation step. Once segments are scored, they can be compared fairly across templates using the ranking approach in Percentile Normalization Across Metric Distributions.

URL Segment Aggregation Pipeline Diagram of five stages: per-URL scores, prefix extraction at a configurable depth, grouping by segment key, a coverage-weighted mean computation, and the resulting segment scores with coverage metadata. Per-URL Scores url, score, sample_count Extract Prefix depth = 2 /blog/2024/post → /blog/2024 Group by Segment pandas groupby() Coverage- Weighted Mean Σ(score × n) ÷ Σ n Segment Scores score, coverage

Environment and Dependencies

The rollup runs against two upstream Parquet exports: the per-URL score table produced by the composite scoring job, and a known-URL-count table derived from the crawl graph so coverage can be computed against what should have been measured, not just what was.

set -euo pipefail
mkdir -p /opt/audit/segment_rollup/{input,output}
cd /opt/audit/segment_rollup
python3 -m venv .venv
.venv/bin/pip install --quiet pandas==2.2.2 pyarrow==16.1.0
export PREFIX_DEPTH=2
export COVERAGE_FLOOR=0.15

PREFIX_DEPTH and COVERAGE_FLOOR are read from the environment rather than hardcoded so the same script runs unchanged for a nightly incremental rollup and a weekly full recompute — only the values that feed the config differ between schedules.

Implementation

The core of this step is a single pandas job: extract a segment key, group, and reduce with a coverage-weighted mean instead of a plain average.

#!/usr/bin/env python3
# /opt/audit/segment_rollup/aggregate_segments.py
"""
Roll per-URL health scores up to path-based segments using a
coverage-weighted mean. Reads PREFIX_DEPTH and COVERAGE_FLOOR from
the environment so the same script serves both the nightly
incremental job and the weekly full recompute without a code change.
"""

import os
from urllib.parse import urlparse

import pandas as pd

# ── 1. environment-injected parameters ────────────────────────────────────
PREFIX_DEPTH = int(os.environ.get("PREFIX_DEPTH", 2))          # /blog/2024/post -> /blog/2024
COVERAGE_FLOOR = float(os.environ.get("COVERAGE_FLOOR", 0.15))  # min measured fraction per segment

# ── 2. load per-URL scores from the upstream composite scoring job ───────
df = pd.read_parquet("/opt/audit/segment_rollup/input/per_url_scores.parquet")
# expected columns: url, score (0-100), sample_count (RUM/crawl samples backing the score)

# ── 3. extract a fixed-depth path segment from each URL ──────────────────
def path_prefix(url: str, depth: int) -> str:
    """Collapse a URL to its first `depth` path components as a stable segment key."""
    parts = [p for p in urlparse(url).path.strip("/").split("/") if p]
    if not parts:
        return "/"
    return "/" + "/".join(parts[:depth])

df["segment"] = df["url"].map(lambda u: path_prefix(u, PREFIX_DEPTH))

# ── 4. total known URLs per segment, sourced from the crawl graph ────────
known = pd.read_parquet("/opt/audit/segment_rollup/input/known_urls_by_segment.parquet")
# expected columns: segment, known_url_count

# ── 5. coverage-weighted mean: influence scales with sample_count, not URL count ─
def weighted_segment(group: pd.DataFrame) -> pd.Series:
    total_samples = group["sample_count"].sum()
    if total_samples == 0:
        return pd.Series({"segment_score": float("nan"), "measured_urls": 0, "total_samples": 0})
    weighted = (group["score"] * group["sample_count"]).sum() / total_samples
    return pd.Series({
        "segment_score": weighted,
        "measured_urls": len(group),
        "total_samples": int(total_samples),
    })

rollup = df.groupby("segment").apply(weighted_segment).reset_index()

# ── 6. join known URL counts and compute a coverage ratio ────────────────
rollup = rollup.merge(known, on="segment", how="left")
rollup["known_url_count"] = rollup["known_url_count"].fillna(rollup["measured_urls"])
rollup["coverage"] = rollup["measured_urls"] / rollup["known_url_count"]

# ── 7. flag low-confidence segments instead of trusting every score equally ──
rollup["low_confidence"] = rollup["coverage"] < COVERAGE_FLOOR

# ── 8. persist segment scores with coverage metadata attached ────────────
out_path = "/opt/audit/segment_rollup/output/segment_scores.parquet"
rollup.sort_values("segment").to_parquet(out_path, index=False)
print(f"Wrote {len(rollup):,} segments -> {out_path}")
print(rollup[rollup.low_confidence][["segment", "coverage"]].to_string(index=False))

A few lines carry the whole design and are worth reading twice. Line 3's path_prefix strips query strings and trailing slashes before truncating, so /blog/2024/post?utm=x and /blog/2024/post/ land in the same segment as /blog/2024/other-post — without that normalization, cosmetically different URLs for the same content type would fragment into separate segments and dilute the sample size behind each one.

Line 5's weighted_segment is the actual coverage-weighted mean: it multiplies each URL's score by its sample_count before summing, then divides by the total sample count for the group. A URL backed by 2,000 RUM sessions pulls the segment score toward it far more than a URL backed by 4 sessions, which is the correct behavior — a score built on more evidence should count for more.

Line 6 is the part a plain groupby().mean() never gives you: it joins against known_url_count, the number of URLs the crawl graph says exist under that segment, not just the number that happened to produce a score. coverage = measured_urls / known_url_count is what lets line 7 distinguish "this segment is genuinely a 91" from "this segment shows 91 because the only three URLs we measured happened to be the fast ones."

Verification and Smoke Test

Run these checks immediately after the job. The first two catch a broken pipeline; the third catches a broken assumption, not a broken script — a script can succeed while quietly returning coverage numbers that make no sense.

set -euo pipefail
cd /opt/audit/segment_rollup

# 1. run the rollup
.venv/bin/python aggregate_segments.py

# 2. output must exist and contain at least one segment
test -s output/segment_scores.parquet && echo "PASS: output written"

# 3. every segment_score must be in [0, 100] or explicitly NaN (no samples)
.venv/bin/python -c "
import pandas as pd
df = pd.read_parquet('output/segment_scores.parquet')
bad = df[(df.segment_score.notna()) & ((df.segment_score < 0) | (df.segment_score > 100))]
print('PASS: all segment scores in range') if bad.empty else print(f'FAIL: {len(bad)} out-of-range segments')
"

# 4. coverage must be in [0, 1] — a ratio above 1 means known_url_count is stale
.venv/bin/python -c "
import pandas as pd
df = pd.read_parquet('output/segment_scores.parquet')
bad = df[(df.coverage < 0) | (df.coverage > 1)]
print('PASS: coverage ratios sane') if bad.empty else print(f'FAIL: {len(bad)} segments with coverage outside [0,1]')
"

# 5. spot-check the flagged low-confidence segments before trusting the rollup
.venv/bin/python -c "
import pandas as pd
df = pd.read_parquet('output/segment_scores.parquet')
print(df[df.low_confidence].sort_values('coverage')[['segment','coverage','measured_urls']].to_string(index=False))
"

Expected output on a healthy run:

Wrote 214 segments -> output/segment_scores.parquet
PASS: output written
PASS: all segment scores in range
PASS: coverage ratios sane
      segment  coverage  measured_urls
/support/legacy      0.04              2

A single low-confidence row like this is normal — it means investigate that segment's number before reporting on it, not that the pipeline failed.

Failure Modes

Sparse segments report a confident-looking score off two or three URLs

A segment with known_url_count = 300 but only 4 measured URLs can still produce a clean-looking segment_score of 88 — the arithmetic is correct, but the confidence behind it is not. Without the coverage floor, that 88 gets treated the same as a segment measured across 280 of 300 URLs.

# diagnostic: list segments below the coverage floor, worst first
.venv/bin/python -c "
import pandas as pd
df = pd.read_parquet('/opt/audit/segment_rollup/output/segment_scores.parquet')
print(df[df.low_confidence].sort_values('coverage')[['segment','coverage','measured_urls','known_url_count']].to_string(index=False))
"
# fix: raise the floor or extend the crawl to increase measured_urls before trusting these scores
export COVERAGE_FLOOR=0.25
.venv/bin/python /opt/audit/segment_rollup/aggregate_segments.py

Prefix collisions merge unrelated URL populations into one segment

Depth-2 truncation means /blog/category-a and /blog/category-b both collapse under /blog only if depth is set to 1 — but the more common real-world version is a legacy path like /blog-archive/2019 accidentally sharing enough structure with /blog/2019 at a chosen depth to be merged by a loose taxonomy check. The two populations have different baseline scores and the merged segment tells you nothing true about either.

# diagnostic: list segment keys with unexpectedly high URL counts
.venv/bin/python -c "
import pandas as pd
df = pd.read_parquet('/opt/audit/segment_rollup/output/segment_scores.parquet')
print(df.sort_values('measured_urls', ascending=False).head(10)[['segment','measured_urls']].to_string(index=False))
"
# fix: validate segment keys against the known URL taxonomy before grouping, not after

Segment keys drift when staging and production URL structures differ

If staging serves content under an extra path component (a locale or version prefix that production strips), the same content type produces different segment keys per environment at a fixed PREFIX_DEPTH. Dashboards built against production segment names silently show gaps when pointed at staging output.

# diagnostic: diff the distinct segment keys between two environments
.venv/bin/python -c "
import pandas as pd
prod = set(pd.read_parquet('output/segment_scores.parquet').segment)
stg  = set(pd.read_parquet('output/segment_scores_staging.parquet').segment)
print('prod-only:', sorted(prod - stg)[:10])
print('staging-only:', sorted(stg - prod)[:10])
"
# fix: strip the environment-specific prefix before path_prefix() runs, not after

FAQ

What prefix depth should I use when segmenting URLs?

Match the depth to a level your team actually acts on, usually 2. Depth 1 collapses an entire site into a handful of top-level buckets and hides where problems live. Depth 3 or deeper fragments a segment into so many near-empty groups that the coverage floor rejects most of them. Start at 2, inspect the low_confidence rate from the aggregation pipeline, and only go deeper for sections with genuinely high URL volume, such as a large product catalog.

How does coverage-weighted mean differ from just averaging scores per segment?

A plain average treats a URL with 3 RUM samples the same as one with 3,000, so a handful of noisy, low-traffic pages can swing a segment's score as much as its highest-traffic page. The coverage-weighted mean multiplies each URL's score by its sample_count before summing and dividing by the total sample count, so influence tracks how much evidence actually backs each score — the same principle used when composite scores are first built in Designing Custom Health Score Algorithms. A separate coverage ratio then tells you how much of the segment was measured at all, which a plain average can never express.

What happens when two different URL patterns collapse into the same segment key?

The groupby has no notion of content type, so a legacy archive path and an unrelated current section that happen to share a prefix at your chosen depth get merged into one segment and one score. This is a prefix collision, and it silently blends two populations with different baselines. Guard against it by validating segment keys against your known URL taxonomy before the rollup runs, and by keeping prefix depth aligned to where your site's structure actually diverges rather than picking one depth for the whole domain.