Keep the last N backups in S3 when Lifecycle cannot

The problem

Plenty of tools write backups to S3 and never clean up: a WordPress backup plugin, a hosting panel's nightly dump, a cron job someone wrote years ago. The bucket grows until someone asks for a retention rule.

The first answer is an S3 Lifecycle rule that expires objects after N days. It is free and built in, and for backups it has two problems.

Lifecycle counts days. "Expire after 7 days" and "keep the last 7 backups" behave the same only while backups arrive every day. When they stop (the plugin is deactivated, the server's disk fills, a credential expires), Lifecycle keeps deleting. Two weeks into a backup outage, an age rule has removed every copy you had. A count rule still holds the last seven backup sets, however old they are. Retention matters most on the bad day, so it needs to survive gaps.

Lifecycle also works per object, and one backup is usually several objects. The two sources in our case each wrote several files per run:

• the plugin wrote six files per run (database, plugins, themes, uploads and two more), sharing a date-time token, each with a hash that changes every run;
• the server backup wrote a .sql and a .tar.gz per night, sharing a trailing 14-digit timestamp, with a middle id that drifts over time.

A rule that deletes per object can split a set: keep the database dump and delete the uploads that go with it.

The job, then, is to group objects into backup sets, order the sets by recency, keep the newest N, delete every object in the rest, and leave alone anything you cannot classify. That fits in a small Lambda function.

The pattern

1. Extract a set key from each object name: identical for every file in one run, different between runs.
2. Plan in a pure function: no AWS calls, the object list in, the keep and delete lists out. This is the part you test.
3. Guard the delete: dry run by default, a circuit breaker on volume, unknown files never touched.
4. Run with a role that can list and delete, and nothing else.
5. Schedule it daily once a dry run has been reviewed.

Step 1: a set key that is stable within a run

The key must not change between the files of one backup, and it must change between backups. The per-run hash in the plugin's filenames fails the first test. The drifting middle id in the server's filenames fails both. The timestamps pass.

A set-key extractor that tries the most specific pattern first:

import re

PLUGIN_RUN = re.compile(r"backup_(\d{4}-\d{2}-\d{2}-\d{4})_")   # plugin date-time token
NIGHTLY_TS = re.compile(r"(?<!\d)(\d{14})(?!\d)")               # YYYYMMDDHHMMSS

def set_key(object_key: str) -> str | None:
    """Return the backup-set id for an object, or None if unrecognized."""
    name = object_key.rsplit("/", 1)[-1]
    if m := PLUGIN_RUN.search(name):
        return f"plugin:{m.group(1)}"
    if m := NIGHTLY_TS.search(name):
        return f"nightly:{m.group(1)}"
    return None

The bare 14-digit pattern is the loosest, so it goes last, and the lookarounds stop it matching inside a longer number. The prefixes keep the two families from colliding. Returning None matters, as Step 3 shows.

Step 2: a pure planner

All the decisions live in one function that takes plain dicts and returns a plan. With no boto3 and no network, you can feed it real filenames copied from the S3 console and assert on the result.

Group, order by the newest object in each set, cut at N:

def plan(objects: list[dict], keep: int) -> dict:
    """objects: [{"Key": str, "LastModified": datetime}, ...]"""
    sets: dict[str, dict] = {}
    unknown: list[str] = []
    for obj in objects:
        sk = set_key(obj["Key"])
        if sk is None:
            unknown.append(obj["Key"])
            continue
        s = sets.setdefault(sk, {"keys": [], "newest": obj["LastModified"]})
        s["keys"].append(obj["Key"])
        s["newest"] = max(s["newest"], obj["LastModified"])

    ranked = sorted(sets.items(), key=lambda kv: (kv[1]["newest"], kv[0]), reverse=True)
    expired = ranked[keep:]
    return {
        "kept": [sk for sk, _ in ranked[:keep]],
        "expired": [sk for sk, _ in expired],
        "delete": [k for _, s in expired for k in s["keys"]],
        "unknown": unknown,
    }

Sets are ordered by the newest LastModified inside them, so a set whose last file landed a minute late still sorts correctly. The tie-break on the set key makes the plan, and so the test, deterministic.

We tested ours offline against the real filenames from both folders: 48 plugin objects in 8 sets, and 40 server objects in 20 nightly sets with the real gaps from the console listing. In both cases it kept the newest 7 sets, deleted every file of the older ones, and left nothing unclassified.

Step 3: guard the delete

A scheduled job that deletes backups needs guards, and they run in this order.

The three outcomes, per folder:

import os

DRY_RUN = os.environ.get("DRY_RUN", "true").lower() != "false"
MAX_DELETE = int(os.environ.get("MAX_DELETE", "0"))   # 0 turns the breaker off

def apply(bucket: str, prefix: str, p: dict) -> dict:
    n = len(p["delete"])
    if MAX_DELETE and n > MAX_DELETE:
        log.error("[%s] %d deletions exceed MAX_DELETE=%d; skipping this folder "
                  "(did the backup filename format change?)", prefix, n, MAX_DELETE)
        return {"blocked": True, "deleted": 0}
    if DRY_RUN:
        for k in p["delete"]:
            log.info("[DRY_RUN] would delete %s", k)
        return {"blocked": False, "deleted": 0}
    return {"blocked": False, "deleted": delete_keys(bucket, p["delete"])}

• Unknown files are never deleted. Anything the extractor cannot classify is logged with a warning and left alone, so a new file type in the bucket cannot be swept up by accident.
• Dry run is the default. A fresh deploy logs [DRY_RUN] would delete ... for every candidate. Going live is a deliberate config change after someone has read that list.
• A circuit breaker on volume. If a folder would lose more objects than MAX_DELETE in one run, deletion for that folder is skipped and an error is logged. The usual cause is a filename format change that makes every file look like its own set. Note the condition: with MAX_DELETE=0, which is the default in this code, the breaker is off. Set it on every deploy, or change the default to a number.

Deletion batches 1,000 keys per request (the API limit) and counts the per-key errors:

def delete_keys(bucket: str, keys: list[str]) -> int:
    deleted = 0
    for i in range(0, len(keys), 1000):
        batch = keys[i:i + 1000]
        resp = s3.delete_objects(
            Bucket=bucket,
            Delete={"Objects": [{"Key": k} for k in batch], "Quiet": True},
        )
        errors = resp.get("Errors", [])
        for e in errors:
            log.error("delete failed %s: %s %s", e.get("Key"), e.get("Code"), e.get("Message"))
        deleted += len(batch) - len(errors)
    return deleted

DeleteObjects returns HTTP 200 even when individual keys fail, so the Errors list in the response is the only place those failures appear.

Step 4: a role that can list and delete, and nothing else

The function does not need to read your backups, so it should not be able to. An inline policy, plus the managed basic execution role for logs:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "List", "Effect": "Allow",
      "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::example-backups" },
    { "Sid": "Delete", "Effect": "Allow",
      "Action": "s3:DeleteObject", "Resource": "arn:aws:s3:::example-backups/*" }
  ]
}

There is no GetObject and no PutObject. If the function's credentials leak, they cannot read a database dump. They can still delete backups, which is the other reason for the circuit breaker.

Step 5: go live in order

1. Deploy with DRY_RUN=true and invoke it once by hand.
2. Read the log. Every would delete line should be a file you expect to lose, and the unknown-file warnings should list only files you expect to keep.
3. Set MAX_DELETE a little above one run's normal volume.
4. Set DRY_RUN=false, invoke once more, and check the summary it returns.
5. Add a daily EventBridge schedule after your backup window, for example cron(0 3 * * ? *).

The traps

Incomplete sets still count. The planner counts backup sets, and it does not check that a set is complete. A run that failed halfway (the .sql uploaded, the .tar.gz did not) is still a set and still takes one of the seven places. A week of half-failed runs would push the last complete sets out. If that matters for your sources, have the planner count the expected files per family (six for the plugin, two for the nightly pair) and rank only complete sets toward N, keeping incomplete ones aside for a person to look at.

A versioned bucket frees nothing. With versioning on, DeleteObject without a version id adds a delete marker, and the old object becomes a noncurrent version that keeps costing money. Check first:

aws s3api get-bucket-versioning --bucket example-backups

If it is on, pair the function with a Lifecycle rule that expires noncurrent versions after a few days and removes expired delete markers (age is fine there, because the count decision has already been made), or extend the planner to list versions and delete by version id.

Order by a clock you trust. LastModified is when the upload finished. If your backup tool can re-upload old files (a resync, a move between buckets), it will misreport their age, and the timestamp in the filename is the better clock.

Loose patterns match more than you expect. A bare \d{14} will match fourteen digits inside a longer number. Keep the specific patterns first, bound the loose ones, prefix the keys by family, and make the offline test on real filenames the gate for any pattern change.

Count per folder. Two sources in one bucket have different cadences. Apply retention per top-level prefix, so a busy source cannot push a quiet one's backups out.

The first run is the slow one. Listing is paginated, so it scales, but a bucket with years of history needs a longer timeout on the first live run than on every run after.

We built this for the same engagement as Running a three-region clinical platform on self-hosted Swarm.