12 min read

Repairing Canonical Tag Conflicts at Scale

Canonical conflicts are the archetypal audit finding that cannot be fixed one page at a time. A canonical declaration is a claim about a relationship between URLs, so the unit that has to be repaired is the group of pages pointing at each other — and a fix applied per page will make each page internally consistent while leaving the group as contradictory as before. This page is part of Writing Remediation Playbooks for Audit Failures and covers repairing them in bulk.

Page by page against as a graph Two columns over the same estate. Fixing page by page makes each page self-consistent, cannot see chains or loops that span pages, produces fixes that conflict with each other, and needs several passes that never converge. Fixing as a graph resolves whole conflict groups at once, sees every shape, produces one internally consistent plan, and converges in a single pass because the target for each group is decided before anything is written. PAGE BY PAGE Each page becomes self-consistent and the estate does not Chains and loops are invisible they span pages by definition Fixes conflict with each other two pages both made canonical Never converges each pass creates new conflicts AS A GRAPH Whole groups resolved at once the unit is the conflict, not the page Every shape is visible chains, loops, dead targets alike One consistent plan the target is chosen once per group Converges in one pass because nothing is decided twice This is the difference between a canonical repair that finishes and one that a team runs every month forever, each time fixing pages that the previous run made inconsistent.

Environment isolation and dependency declaration 

set -euo pipefail
export CAN_EXPORT="/data/crawl/latest/pages.jsonl"
export CAN_TRAFFIC="/data/analytics/sessions_90d.parquet"   # the tie-break
export CAN_PLAN_OUT="/data/remediation/canonical_plan.tsv"
export CAN_MAX_GROUP=50            # refuse to auto-resolve a huge group
export CAN_DRY_RUN=1
/opt/audit/.venv/bin/pip install "networkx==3.3" "pandas==2.2.2"

CAN_MAX_GROUP exists because conflict group size is the best available proxy for "this is a template problem, not a page problem". A group of four pages is a content mistake; a group of nine hundred is a template emitting the wrong canonical, and rewriting nine hundred pages is the wrong fix for it.

Four conflicts, four different fixes Four rows. A self-referencing chain, where A declares B canonical and B declares C, means the intended target is ambiguous and the chain must be collapsed to one hop. A conflicting pair, where two pages each declare the other, is a loop nobody can resolve and one side must be chosen. A canonical pointing at a non-200 URL declares a target that does not resolve. And a canonical that disagrees with the sitemap means two signals contradict each other and the sitemap should follow the page. CONFLICT SHAPE WHAT IT DOES FIX A chain of canonicals A declares B, B declares C, C declares itself The target is ambiguous a consumer may stop at B or follow through to C Collapse to one hop every member declares C directly A conflicting pair A declares B canonical and B declares A An unresolvable loop neither page can be preferred over the other Choose one side from traffic and inbound links, then make it self-referencing Canonical points at a URL that 404s or redirects elsewhere The target does not resolve the declaration asserts a page that is not there Repoint or remove to the resolved destination, or to self Canonical and sitemap disagree about which URL is preferred Two contradictory signals consumers weight them differently Sitemap follows the page regenerate the sitemap from the canonical set Only the first two are visible from the page alone. The third and fourth need the crawl graph and the sitemap, which is why canonical repair belongs to the audit rather than to a page-level linter.

Implementation 

#!/usr/bin/env python3
# /opt/audit/heal/canonical.py
# Repair canonicals as a graph: group, choose one target per group, emit a plan.
from __future__ import annotations
import json
import os
import sys
import networkx as nx
import pandas as pd

MAX_GROUP = int(os.environ.get("CAN_MAX_GROUP", "50"))


def build_graph(export_path: str) -> tuple[nx.DiGraph, dict]:
    g, pages = nx.DiGraph(), {}
    with open(export_path, encoding="utf-8") as fh:
        for line in fh:
            rec = json.loads(line)
            url, canon = rec["final_url"], rec.get("canonical_url")
            pages[url] = rec
            g.add_node(url)
            if canon and canon != url:
                g.add_edge(url, canon)          # declared, resolved upstream
    return g, pages


def classify(g: nx.DiGraph, pages: dict) -> dict[str, str]:
    kinds = {}
    for u, v in g.edges():
        target = pages.get(v)
        if target is None:
            kinds[u] = "target_not_crawled"
        elif target.get("status") != 200:
            kinds[u] = "target_not_200"
        elif g.has_edge(v, u):
            kinds[u] = "conflicting_pair"
        elif g.out_degree(v) > 0:
            kinds[u] = "canonical_chain"
    return kinds


def choose_target(group: set[str], traffic: pd.Series, pages: dict) -> str:
    """The one judgement in the whole process. Prefer, in order: the page
    with the most sessions, then the most inbound internal links, then the
    shortest path — a deterministic tie-break so two runs agree."""
    ranked = sorted(
        group,
        key=lambda u: (-float(traffic.get(u, 0.0)),
                       -int(pages.get(u, {}).get("inlinks", 0)),
                       len(u)))
    return ranked[0]


def plan(export_path: str, traffic_path: str) -> list[tuple[str, str, str]]:
    g, pages = build_graph(export_path)
    kinds = classify(g, pages)
    traffic = (pd.read_parquet(traffic_path).set_index("url")["sessions"]
               if os.path.exists(traffic_path) else pd.Series(dtype=float))

    rows: list[tuple[str, str, str]] = []
    for group in nx.weakly_connected_components(g):
        if len(group) < 2:
            continue
        if len(group) > MAX_GROUP:
            print(f"SKIP: group of {len(group)} — treat as a template fault",
                  file=sys.stderr)
            continue
        target = choose_target(group, traffic, pages)
        for url in sorted(group):
            if pages.get(url, {}).get("canonical_url") != target:
                rows.append((url, target, kinds.get(url, "group_alignment")))
    return rows

The decisions that carry it:

  • The unit is a weakly connected component, not a page. Every URL that participates in the same tangle is resolved together, which is what makes the pass converge instead of shuffling conflicts around.
  • choose_target has a deterministic tie-break all the way down. Two runs over the same data must produce the same plan, or a dry-run diff cannot be approved and applied later.
  • Groups above the cap are skipped with a message rather than resolved. A group of hundreds is a template emitting the wrong canonical, and the correct fix is one template change, not hundreds of page rewrites.
  • The plan is emitted, not applied. Canonical rewrites touch templates and CMS fields, so the output goes to review exactly as the parent playbook pattern requires.
Four stages of a bulk repair Four stages left to right. Build the canonical graph across the whole export, resolving every declared target through its redirects. Classify each node into one of the four conflict shapes. Choose a target per conflict group using traffic and inbound links as the tie-break. Then emit a rewrite plan and regenerate the sitemap so both signals agree. Build the graph resolve every declared target Classify chain, pair, dead, disagree Choose a target traffic and inbound links Emit and align rewrite plan + new sitemap Choosing the target is the only stage requiring judgement, which is why it is isolated: everything before it is mechanical and everything after it follows from the choice.

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
import json, tempfile, os
from canonical import plan

recs = [
  {"final_url": "https://x.test/a", "status": 200, "canonical_url": "https://x.test/b", "inlinks": 5},
  {"final_url": "https://x.test/b", "status": 200, "canonical_url": "https://x.test/a", "inlinks": 40},
  {"final_url": "https://x.test/c", "status": 200, "canonical_url": "https://x.test/c", "inlinks": 2},
]
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
    for r in recs: fh.write(json.dumps(r) + "\n")
    src = fh.name

rows = plan(src, "/nonexistent.parquet")
targets = {t for _, t, _ in rows}
assert targets == {"https://x.test/b"}, targets      # more inlinks wins
assert all(u != "https://x.test/c" for u, _, _ in rows)  # untangled page untouched
print(f"PASS: pair resolved to {targets.pop()}, self-canonical page untouched")

# Determinism: two runs must agree exactly.
assert plan(src, "/nonexistent.parquet") == rows
print("PASS: the plan is deterministic")
os.unlink(src)
PY

Expected output is two PASS lines. The determinism assertion is the one that makes the plan reviewable — an approval granted on Monday is worthless if Thursday's run produces a different plan from identical inputs.

Failure modes

The plan rewrites more pages than the finding covered

A group grew because one page in it declares a canonical outside the intended scope, pulling an unrelated cluster of URLs into the same component. Cap the group size, and inspect any group that approaches the cap before approving it.

Two runs produce different plans

A tie-break is non-deterministic — most often a set iteration order or a traffic lookup that returns different values because the analytics window moved. Pin the traffic window to a fixed date range for the duration of a plan-and-apply cycle, and sort every collection before choosing.

The sitemap still disagrees after the rewrite

Regenerating the sitemap is a separate step and is easy to forget. The canonical set is the source of truth; regenerate the sitemap from it in the same change, and reconcile the result with resolving orphaned sitemap entries so the two signals cannot drift apart again.

FAQ

Why can canonical conflicts not be fixed page by page?

Because a canonical is a claim about a relationship between URLs, so the smallest repairable unit is the group of pages that point at each other. Fixing one page makes that page internally consistent and leaves the group as contradictory as before — and because each pass creates new inconsistencies in pages it did not touch, the process never converges. Resolving whole connected components fixes each tangle once.

How should the surviving canonical target be chosen?

By a deterministic ranking rather than a judgement made per group. Sessions over a fixed window is the strongest signal, inbound internal links the next, and something arbitrary but stable such as URL length as the final tie-break. The determinism matters as much as the ordering: a plan approved on one day and applied later must be regenerable identically, and a tie-break that depends on set iteration order or a moving analytics window makes that impossible.

What does a very large conflict group indicate?

A template fault rather than a content mistake. Four pages tangled together is somebody setting a canonical by hand incorrectly; nine hundred pages sharing one conflict is a template emitting the wrong value, and rewriting nine hundred pages is both the expensive fix and the one that will be undone the next time the template renders. Cap the group size the repair will resolve automatically and route anything above it to the team that owns the template.