12 min read

Expiring Old Crawl Artifacts with Lifecycle Rules

A crawl pipeline that writes a fresh snapshot every night accumulates storage cost linearly forever unless something prunes it. This page is part of Storing & Versioning Crawl Artifacts in Cloud Storage and covers the specific mechanism for doing that pruning safely: declarative object-storage lifecycle rules on Google Cloud Storage (GCS) and Amazon S3 that tier aging artifacts to cheaper storage classes and expire them by age, without ever touching the checksummed baseline objects that later audits diff against.

The failure mode this guide exists to prevent is a lifecycle rule that is too broad. A rule scoped to the whole bucket instead of a daily/ prefix will eventually reach into baselines/ and delete the one artifact every regression comparison depends on. Getting the prefix scoping and the noncurrent-version handling right the first time avoids a very quiet, very expensive mistake.

Crawl Artifact Lifecycle Tiering Diagram showing daily crawl artifacts under the daily/ prefix moving through four age-gated stages — Standard, Nearline/IA, Coldline/Glacier, and Delete — while objects under the baselines/ prefix are shown on a separate branch marked as excluded from all lifecycle transitions. daily/ Standard age 0–30d Nearline / IA age 30–90d Coldline / Glacier age 90–365d Delete age > 365d baselines/ — excluded from every lifecycle rule retained indefinitely, protected by Object Lock / retention policy never matches baselines/ prefix

Prerequisites and Environment

You need write access to lifecycle configuration on both buckets, gcloud and aws CLIs authenticated to the right project and account, and — critically — agreement across the team on which prefix holds disposable daily artifacts versus which prefix holds immutable baselines. This split has to exist before you write a single rule.

set -euo pipefail

# pinned CLI versions used in this guide
gcloud version | grep "Google Cloud SDK" # Google Cloud SDK 480.0.0
aws --version                            # aws-cli/2.17.corresponding

export GCS_BUCKET="audit-crawl-artifacts-prod"
export S3_BUCKET="audit-crawl-artifacts-prod"
export AWS_REGION="us-east-1"

# expected prefix layout in both buckets — enforce this before applying rules
# daily/YYYY-MM-DD/*      — disposable nightly snapshots, safe to expire
# baselines/*             — checksummed, immutable, never touched by lifecycle

Implementation

The two policies below are deliberately symmetric: 30 days in the hot tier, a mid-tier hop at 90 days, a cold tier out to 365 days, then deletion — but only for objects under daily/. Neither document contains a rule that can match baselines/.

{
  "//": "gcs-lifecycle.json — applies only to gs://audit-crawl-artifacts-prod",
  "//1": "GCS evaluates rules top-to-bottom against every object daily.",
  "rule": [
    {
      "//action": "move daily/ objects to Nearline once they age past 30 days",
      "action": { "type": "SetStorageClass", "storageClass": "NEARLINE" },
      "condition": {
        "age": 30,
        "matchesPrefix": ["daily/"],
        "matchesStorageClass": ["STANDARD"]
      }
    },
    {
      "//action": "move daily/ Nearline objects to Coldline at 90 days",
      "action": { "type": "SetStorageClass", "storageClass": "COLDLINE" },
      "condition": {
        "age": 90,
        "matchesPrefix": ["daily/"],
        "matchesStorageClass": ["NEARLINE"]
      }
    },
    {
      "//action": "delete daily/ objects entirely once they pass 365 days",
      "action": { "type": "Delete" },
      "condition": {
        "age": 365,
        "matchesPrefix": ["daily/"]
      }
    },
    {
      "//action": "purge noncurrent versions of daily/ objects after 60 days to cap version sprawl",
      "action": { "type": "Delete" },
      "condition": {
        "daysSinceNoncurrentTime": 60,
        "matchesPrefix": ["daily/"],
        "isLive": false
      }
    }
  ]
}
{
  "//": "s3-lifecycle.json — applies only to bucket audit-crawl-artifacts-prod",
  "Rules": [
    {
      "ID": "daily-tier-and-expire",
      "Status": "Enabled",
      "Filter": { "Prefix": "daily/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 },
      "NoncurrentVersionTransitions": [
        { "NoncurrentDays": 30, "StorageClass": "STANDARD_IA" }
      ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 60 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "baselines-no-expiry",
      "Status": "Enabled",
      "Filter": { "Prefix": "baselines/" },
      "Transitions": [
        { "Days": 90, "StorageClass": "GLACIER" }
      ]
    }
  ]
}

Line-by-line, the parts that matter most:

  • matchesPrefix: ["daily/"] (GCS) and Filter.Prefix: "daily/" (S3) are the guardrail — every destructive condition is scoped to this prefix only. The baselines-no-expiry S3 rule shows the pattern for a prefix you want tiered for cost but never expired: it has Transitions and no Expiration block at all.
  • matchesStorageClass on the GCS rules prevents a rule from re-firing on an object it already moved — without it, the Nearline-to-Coldline condition could also incorrectly re-evaluate Standard objects that happen to be 90+ days old and skip straight past the Nearline step, which changes your cost curve.
  • NoncurrentVersionExpiration / daysSinceNoncurrentTime targets prior versions of an object, not the current one. This is what actually caps cost in a versioned bucket — the artifact storage and versioning guide covers why versioning is turned on in the first place, and this rule is the other half of that decision: versioning without a noncurrent-expiry rule grows storage forever.
  • AbortIncompleteMultipartUpload cleans up partial uploads left behind by a crashed crawl-artifact push, which otherwise bill for storage but never surface as a listable object.

Apply both documents:

set -euo pipefail

# GCS — bucket-level lifecycle update
gcloud storage buckets update "gs://${GCS_BUCKET}" \
  --lifecycle-file=gcs-lifecycle.json

# S3 — bucket-level lifecycle update
aws s3api put-bucket-lifecycle-configuration \
  --bucket "${S3_BUCKET}" \
  --lifecycle-configuration file://s3-lifecycle.json \
  --region "${AWS_REGION}"

Verification

Read the policy back from both providers and confirm the baselines/ prefix carries no expiration rule before you consider the change complete.

set -euo pipefail

# 1. confirm GCS accepted the policy and it matches what you pushed
gcloud storage buckets describe "gs://${GCS_BUCKET}" --format="json(lifecycle)" \
  | tee /tmp/gcs-applied-lifecycle.json
diff <(jq -S . gcs-lifecycle.json) <(jq -S .lifecycle /tmp/gcs-applied-lifecycle.json) \
  && echo "PASS: GCS lifecycle matches source" || echo "FAIL: GCS lifecycle drifted"

# 2. confirm S3 accepted the policy
aws s3api get-bucket-lifecycle-configuration --bucket "${S3_BUCKET}" \
  | tee /tmp/s3-applied-lifecycle.json
jq -e '.Rules[] | select(.Filter.Prefix=="baselines/") | has("Expiration") | not' \
  /tmp/s3-applied-lifecycle.json \
  && echo "PASS: baselines/ has no Expiration rule" \
  || echo "FAIL: baselines/ is expirable — fix before next nightly run"

# 3. dry-run: list objects that WOULD be deleted under the daily/ rule right now
aws s3api list-objects-v2 --bucket "${S3_BUCKET}" --prefix "daily/" \
  --query "Contents[?LastModified<='$(date -u -d '365 days ago' +%Y-%m-%dT%H:%M:%S)'].Key" \
  --output text

Expected output on a healthy policy:

PASS: GCS lifecycle matches source
PASS: baselines/ has no Expiration rule

An empty result from step 3 on a fresh bucket is expected — nothing should be old enough to expire yet. Re-run it after the first full year of nightly crawls to confirm the rule is actually being evaluated by the provider's own daily scan, not just accepted by the API.

Failure Modes

A broad rule accidentally matches the baseline prefix

This happens when someone edits the lifecycle document by hand and drops the Filter.Prefix / matchesPrefix condition "to simplify it." The rule then matches every object in the bucket, including baselines/.

# diagnostic: any rule with an Expiration/Delete action and no baselines/ exclusion?
jq '.Rules[] | select(.Expiration != null and (.Filter.Prefix // "") != "daily/")' \
  s3-lifecycle.json
# fix: re-apply the known-good document from version control
aws s3api put-bucket-lifecycle-configuration \
  --bucket "${S3_BUCKET}" --lifecycle-configuration file://s3-lifecycle.json

Noncurrent version expiration deletes versions still needed for a rollback window

If NoncurrentVersionExpiration is set too aggressively (say, 7 days) relative to how long you actually need to roll back to a previous night's crawl, a hotfix rollback three weeks later finds the version already gone.

# diagnostic: list remaining noncurrent versions for a specific key
aws s3api list-object-versions --bucket "${S3_BUCKET}" \
  --prefix "daily/2026-06-01/" \
  --query "Versions[?IsLatest==\`false\`].[Key,VersionId,LastModified]" --output table
# fix: widen the noncurrent-day window and re-apply
# edit NoncurrentVersionExpiration.NoncurrentDays to a safer value, then re-run put-bucket-lifecycle-configuration

A GCS retention policy or Object Lock conflicts with a transition rule

If baselines/ is under a bucket-level retention policy or an object is under Object Lock, a SetStorageClass transition can still apply, but a Delete action targeting the same object silently fails and the object is never removed even after the intended condition is met — which is correct behavior for a baseline, but confusing if you expected an unrelated cleanup rule to also apply there.

# diagnostic: check for an active retention policy on the bucket
gcloud storage buckets describe "gs://${GCS_BUCKET}" --format="json(retentionPolicy)"
# fix: nothing to fix if this is the baselines/ prefix — this is the guardrail working as intended.
# if it's blocking an intended cleanup on a non-baseline prefix, shorten or remove the retention policy scope.

FAQ

Will lifecycle rules delete my checksummed baseline snapshots?

Only if you let a rule's prefix match the baseline path. Keep baselines under a dedicated prefix such as baselines/ that never appears in an Age or Delete condition, and add a bucket-level retention policy or Object Lock on that prefix so a misconfigured rule cannot remove it even by accident.

Does GCS support the same Nearline and Coldline tiers as S3 has Glacier?

The concepts map but the names and minimum storage durations differ. GCS offers Standard, Nearline (30-day minimum), Coldline (90-day minimum), and Archive (365-day minimum). S3 offers Standard, Standard-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive, each with its own minimum duration charge. Match your transition ages to each provider's minimum or you will pay an early-deletion penalty.

How do I stop old object versions from silently accumulating cost?

Enable NoncurrentVersionExpiration on S3 (or the equivalent isLive: false condition on GCS) with a bounded noncurrent-day window, for example 60 days. Without it, a versioned bucket keeps every prior upload indefinitely and storage cost grows unbounded even though only the latest object is ever read. This is the same versioning trade-off discussed in deduplicating URLs in a distributed crawl, where unbounded state growth is the recurring risk to design against.

Can I apply a lifecycle rule retroactively to objects already in the bucket?

Yes. Both GCS and S3 evaluate lifecycle rules against all existing objects the next time the daily lifecycle scan runs, not just newly written ones. There is no backfill command to trigger early; the scan runs on the provider's own schedule, typically within 24 hours of the rule being applied.