13 min read

Resolving Orphaned Sitemap Entries

A sitemap that has drifted from reality is worse than no sitemap at all — it tells crawlers to spend budget on URLs that 404, redirect, or carry a canonical tag pointing somewhere else entirely, while staying silent about pages that actually deserve discovery priority. This happens quietly: templates get retired, canonical rules change, redirects get added for a migration and never get reflected upstream in the sitemap generator. This guide is part of Writing Remediation Playbooks for Audit Failures and covers the specific playbook for reconciling sitemap.xml against the live crawl graph — diffing the two sets, classifying every mismatch by reason, and emitting reviewable prune and add lists instead of rewriting the sitemap blind.

The reconciliation only works if both sides of the comparison are trustworthy. The crawl graph has to reflect actual reachability — which URLs a crawler starting from the homepage can actually reach, at what depth, and with what final status — the same graph produced by the process described in mapping URL hierarchies before running a crawl. A sitemap diffed against a stale or partial crawl graph will misclassify perfectly healthy URLs as orphaned, so the freshness of the crawl export matters as much as the sitemap parsing itself.

Sitemap vs Crawl Graph Reconciliation Sitemap.xml is parsed into a declared URL set and the crawl graph is parsed into a reachable canonical URL set. Both sets feed a diff engine, which outputs a prune list of orphaned sitemap entries and an add list of missing indexable URLs. sitemap.xml declared URL set Crawl graph reachable + canonical Diff engine normalize + compare Prune list orphaned entries Add list missing indexable URLs

Implementation

The script below does one job: read a sitemap (plain or gzipped), read a crawl graph export, normalize both, and emit two CSV files — prune_list.csv for sitemap entries the crawl graph cannot back up, and add_list.csv for indexable URLs the crawl graph found but the sitemap never declared. It intentionally does not touch sitemap.xml directly; the output is a reviewable diff, not an automatic rewrite, because pruning the wrong URL from a sitemap is a slower, quieter mistake than most audit failures.

#!/usr/bin/env python3
# /opt/audit/sitemap_reconcile/reconcile_sitemap.py
"""
Diff sitemap.xml against a crawl-graph export and emit prune/add lists.
Reads paths from environment so the script is identical across environments.
"""

import csv
import gzip
import os
import sys
import json
from urllib.parse import urlsplit, urlunsplit
from xml.etree import ElementTree as ET

SITEMAP_PATH = os.environ.get("SITEMAP_PATH", "/opt/audit/sitemap_reconcile/input/sitemap.xml")
CRAWL_GRAPH_PATH = os.environ.get("CRAWL_GRAPH_PATH", "/opt/audit/sitemap_reconcile/input/crawl_graph.jsonl")
OUT_DIR = os.environ.get("OUT_DIR", "/opt/audit/sitemap_reconcile/output")
SITEMAP_NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

# ── 1. normalize a URL so trivial variants never register as a mismatch ──────
def normalize(url: str) -> str:
    parts = urlsplit(url.strip())
    scheme = "https"                       # canonicalize scheme, this site is https-only
    netloc = parts.netloc.lower()          # host casing is not significant
    path = parts.path
    if path != "/" and path.endswith("/"):
        path = path.rstrip("/")            # trailing slash is not significant on this site
    return urlunsplit((scheme, netloc, path, "", ""))  # drop query + fragment for the diff

# ── 2. read sitemap.xml transparently, whether gzipped or plain ─────────────
def load_sitemap(path: str) -> dict:
    opener = gzip.open if path.endswith(".gz") else open
    with opener(path, "rb") as fh:
        root = ET.fromstring(fh.read())
    urls = {}
    for url_el in root.findall(f"{SITEMAP_NS}url"):
        loc_el = url_el.find(f"{SITEMAP_NS}loc")
        if loc_el is None or not loc_el.text:
            continue
        lastmod_el = url_el.find(f"{SITEMAP_NS}lastmod")
        urls[normalize(loc_el.text)] = {
            "raw_loc": loc_el.text.strip(),
            "lastmod": lastmod_el.text.strip() if lastmod_el is not None else None,
        }
    return urls

# ── 3. read the crawl graph export (one JSON object per reached URL) ────────
def load_crawl_graph(path: str) -> dict:
    reachable = {}
    with open(path, "r", encoding="utf-8") as fh:
        for line in fh:
            if not line.strip():
                continue
            row = json.loads(line)
            # expected keys: url, status, canonical_url, indexable, crawled_at
            reachable[normalize(row["url"])] = row
    return reachable

# ── 4. classify every sitemap URL against the crawl graph ───────────────────
def diff(sitemap_urls: dict, crawl_graph: dict):
    prune, add = [], []

    for norm_url, entry in sitemap_urls.items():
        row = crawl_graph.get(norm_url)
        if row is None:
            prune.append((entry["raw_loc"], "unreached_by_crawl"))
        elif row["status"] >= 400:
            prune.append((entry["raw_loc"], f"http_{row['status']}"))
        elif row["status"] in (301, 302, 307, 308):
            prune.append((entry["raw_loc"], "redirects"))
        elif normalize(row.get("canonical_url", norm_url)) != norm_url:
            prune.append((entry["raw_loc"], "non_canonical"))
        elif not row.get("indexable", True):
            prune.append((entry["raw_loc"], "noindex"))

    for norm_url, row in crawl_graph.items():
        eligible = (
            row["status"] == 200
            and row.get("indexable", True)
            and normalize(row.get("canonical_url", norm_url)) == norm_url
        )
        if eligible and norm_url not in sitemap_urls:
            add.append((row["url"], "reachable_not_declared"))

    return prune, add

# ── 5. write reviewable CSV artifacts, never touch sitemap.xml directly ─────
def write_csv(path: str, rows: list, header: tuple):
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow(header)
        writer.writerows(rows)

def main():
    sitemap_urls = load_sitemap(SITEMAP_PATH)
    crawl_graph = load_crawl_graph(CRAWL_GRAPH_PATH)
    prune, add = diff(sitemap_urls, crawl_graph)

    write_csv(f"{OUT_DIR}/prune_list.csv", prune, ("url", "reason"))
    write_csv(f"{OUT_DIR}/add_list.csv", add, ("url", "reason"))

    print(f"sitemap entries: {len(sitemap_urls):,}")
    print(f"crawl graph entries: {len(crawl_graph):,}")
    print(f"prune candidates: {len(prune):,} -> {OUT_DIR}/prune_list.csv")
    print(f"add candidates: {len(add):,} -> {OUT_DIR}/add_list.csv")

if __name__ == "__main__":
    sys.exit(main())

A few lines carry the whole design:

  • Step 1 (normalize) collapses scheme, host casing, and trailing-slash variants before either set is built, so https://Example.com/blog/post/ and https://example.com/blog/post count as the same URL instead of manufacturing a false mismatch on both sides of the diff.
  • Step 2 (load_sitemap) picks gzip.open or open based on the file extension, so a .xml.gz sitemap parses the same way a plain one does — no separate code path to keep in sync.
  • Step 3 (load_crawl_graph) expects one JSON object per line with status, canonical_url, and indexable, which is exactly the shape produced by the crawl-graph export described in mapping URL hierarchies before running a crawl.
  • Step 4 (diff) classifies every prune candidate with a reason code instead of a bare true/false — unreached_by_crawl, http_4xx/http_5xx, redirects, non_canonical, noindex — because each reason implies a different fix, and the redirect case overlaps directly with the workflow in fixing broken redirect chains.
  • Step 5 never mutates sitemap.xml. It writes CSVs a human or a follow-up CI step reviews before regenerating the sitemap, keeping the destructive action out of this script entirely.

Run it with:

set -euo pipefail
export SITEMAP_PATH=/opt/audit/sitemap_reconcile/input/sitemap.xml
export CRAWL_GRAPH_PATH=/opt/audit/sitemap_reconcile/input/crawl_graph.jsonl
export OUT_DIR=/opt/audit/sitemap_reconcile/output
python3 /opt/audit/sitemap_reconcile/reconcile_sitemap.py

Verification

Treat the diff output as a candidate list, not a finished action, until it passes three checks:

set -euo pipefail
cd /opt/audit/sitemap_reconcile

# 1. both output files exist and are non-empty (headers at minimum)
test -s output/prune_list.csv && echo "PASS: prune_list.csv written"
test -s output/add_list.csv && echo "PASS: add_list.csv written"

# 2. prune count is a small fraction of total sitemap size — a large jump
#    usually means the crawl graph is stale or partial, not that the site broke
SITEMAP_TOTAL=$(($(wc -l < input/sitemap.xml) ))
PRUNE_COUNT=$(($(wc -l < output/prune_list.csv) - 1))
echo "prune candidates: $PRUNE_COUNT"
[ "$PRUNE_COUNT" -lt 500 ] && echo "PASS: prune count within expected bound" \
  || echo "WARN: prune count unusually high — check crawl graph freshness first"

# 3. spot-check five prune candidates against the live site before removing anything
tail -n +2 output/prune_list.csv | shuf | head -n 5 | cut -d, -f1 | while read -r url; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -L "$url")
  echo "$url -> $code"
done

Expected output looks like:

PASS: prune_list.csv written
PASS: add_list.csv written
prune candidates: 41
PASS: prune count within expected bound
https://example.com/old-campaign-landing/ -> 404
https://example.com/legacy/pricing-2023/ -> 301

If a spot-checked URL comes back 200 when the crawl graph flagged it, the crawl graph is out of date relative to the live site — regenerate the crawl export and re-run the diff before touching the sitemap. Only after the spot check confirms the reasons hold up should the prune list feed the sitemap generator's exclusion rules on the next build.

Failure Modes

Sitemap index files silently produce an empty or near-empty diff

A sitemap_index.xml file contains <sitemap><loc> entries pointing at child sitemaps, not <url><loc> entries. Running load_sitemap directly against the index returns zero URLs, and the diff reports every reachable page as "missing" — which looks like a bug in the crawl graph but is actually a bug in what got parsed.

# diagnostic: check which top-level tag the file actually uses
grep -o -m1 -E "<sitemapindex|<urlset" input/sitemap.xml

# fix: dereference the index first, then concatenate child <url> entries
python3 -c "
from xml.etree import ElementTree as ET
NS = '{http://www.sitemaps.org/schemas/sitemap/0.9}'
root = ET.parse('input/sitemap.xml').getroot()
for sm in root.findall(f'{NS}sitemap'):
    print(sm.find(f'{NS}loc').text)
" | while read -r child_url; do
  curl -s "$child_url" -o "input/children/$(basename "$child_url")"
done

Gzipped sitemaps read as plain text corrupt the parse

Some CDNs serve sitemap.xml.gz with a .xml extension and a Content-Encoding: gzip header, or vice versa — a .gz filename that is not actually gzip-compressed at rest. Branching only on file extension, as the script above does for simplicity, breaks in either direction.

# diagnostic: check the actual file magic bytes, not the extension
file input/sitemap.xml.gz
# gzip compressed data -> extension matches content, proceed
# ASCII text -> extension lies, strip .gz handling and parse as plain XML

# fix: sniff magic bytes instead of trusting the extension
python3 -c "
with open('input/sitemap.xml.gz', 'rb') as f:
    magic = f.read(2)
print('gzip' if magic == b'\x1f\x8b' else 'plain')
"

lastmod drift makes recently-published pages look orphaned

If the crawl graph export is a day or more older than the sitemap's newest lastmod timestamps, brand-new pages the sitemap correctly declares will show up as unreached_by_crawl simply because the crawl hasn't gotten to them yet — not because they are actually orphaned.

# diagnostic: compare crawl export freshness against the newest sitemap lastmod
python3 -c "
import json
from datetime import datetime, timezone
newest_lastmod = max(
    datetime.fromisoformat(l) for l in ['2026-07-04T00:00:00+00:00']
)
crawl_ts = datetime.fromtimestamp(1751500000, tz=timezone.utc)  # crawl_graph.jsonl mtime
print('crawl graph is stale relative to sitemap' if crawl_ts < newest_lastmod else 'fresh')
"

# fix: re-run the crawl before diffing, or exclude URLs newer than the crawl window

FAQ

What counts as an orphaned sitemap entry?

A sitemap URL is orphaned when the crawl graph shows it returns a 4xx or 5xx status, redirects somewhere else, carries a canonical tag pointing at a different URL, or was never reached by the crawl despite being linked nowhere on the site. Any of those four conditions means the sitemap is asserting an index signal the live site does not back up.

Should I remove sitemap entries that are only temporarily unreachable?

No. Require the crawl graph to confirm unreachability across at least two consecutive crawl runs before pruning. A single failed fetch is often a transient timeout, a rate-limit response, or a deploy-window blip, and pruning on one data point produces false positives that then have to be re-added.

How do I handle a sitemap index that references multiple child sitemaps?

Dereference the sitemap index first: fetch every <loc> under <sitemap> and parse each child document's <url> entries into one combined URL set before diffing. Diffing only the index file itself silently produces an empty or near-empty comparison because the index has no <url> entries of its own.

Does removing an orphaned URL from the sitemap deindex it?

No, removing a URL from the sitemap only stops asserting it as a discovery and priority signal; it does not remove an already-indexed page. If the goal is deindexing, the URL needs a noindex directive or a 410/404 response and time for re-crawling, independent of the sitemap prune.