A nightly batch pipeline that cannot report an empty run as success

The situation

The pipeline collects and scores sales leads for our consulting work. It runs every night as a series of one-shot Docker containers under a systemd timer, on a small VM, and writes to a managed Postgres database that also sits behind our web app.

By August 2026 nothing was crashing, but the run summary printed "8 scrapers, 0 failed" on nights when two sources had returned nothing, and runtime was climbing toward the cap on the systemd unit. We had also decided to develop the web app against production data, backed up and restore-verified first and guarded by count checks, so every schema change was a change to live rows.

What we did

Empty runs that looked healthy

Scrapers returned zero rows for nights at a time while the run reported success. We fixed it three times, and each fix named one more specific cause: a feed that changed shape, a URL rule that collapsed rows, a source that had been dead for months. Then a fourth empty run appeared: an ingest step fetched 5 items, mapped 0, and recorded no error.

The mapping function had three ways to drop an item, and the caller that explained an empty result knew about one. So we put an invariant where the rows are dropped: the mapper counts each drop by reason, and if items came in and none survived, the run record in the database gets the breakdown. A partial drop stays unflagged, because some attrition is normal. An illustration of the shape:

def map_batch(items, mapper, run):
    drops: dict[str, int] = {}
    kept = []
    for item in items:
        row = mapper(item, drops)          # the mapper records why it dropped
        if row is not None:
            kept.append(row)
    # Items arrived and none survived: this must never pass as a success.
    if items and not kept:
        reasons = " ".join(f"{k}={v}" for k, v in sorted(drops.items())) or "unknown"
        run.zero_reason = f"fetched={len(items)} kept=0 {reasons}"
        run.failed = True                  # stored on the run row, read by the summary
    return kept

The summary had a second gap. It counted a scraper that raised an exception as failed, but one that caught its own error and reported it in a structured list as healthy. From August 27, two sources returned zero every night, and for about a week each night printed "0 failed". Both shapes now fail the stage, while the systemd unit's own failure stays reserved for "the pipeline did not run".

A runtime creeping toward its cap

On September 2 the run took 2h43m45s, and 1h50m of it was one stage: revalidation, which re-checks whether older postings are still open. When a source blocks the check, the answer is "unknown", and by design an unknown answer does not close a posting. But nothing limited how often a blocked source was asked. One source had 1,758 attempts and no verdict, ever. Another had started serving a managed bot challenge, so each failed check cost about five minutes; with 187 checks pending, that was 15.6 hours of work that could not succeed.

We added a per-source circuit breaker with two arms. One counts consecutive unknowns; the other adds up the seconds they cost, because the count alone tripped in about 5 seconds on one source and after 31m54s on another. Any real verdict resets both, so a source that recovers is back the next night.

The same week, unattended upgrades restarted the service 1h47m into a healthy run. It started again from stage one and hit the cap, and the killed run left job containers holding most of the VM's memory. Each stage now has its own wall-clock budget, summing to less than a raised five-hour cap, and the runner cleans up after itself:

reap() { docker compose --profile jobs rm -fsv >/dev/null 2>&1 || true; }
on_signal() { trap - EXIT TERM INT; reap; kill -s "$1" "$$"; }   # re-raise for systemd
trap reap EXIT
trap 'on_signal TERM' TERM
trap 'on_signal INT' INT

for stage in $STAGES; do
  if ! timeout -k 60s "$(budget_for "$stage")" \
       docker compose --profile jobs run --rm "$stage"; then
    failed=$((failed + 1))     # exit 124 or 137: the stage ran out of budget
  fi
done

The upgrade helper got a narrow exemption for this one unit, so SSH and Docker still restart after security updates.

Schema changes on production data

The scraper deploy applies migrations when a schema change reaches the production branch, so on September 21 a merge applied one that we had described as held back. Four days later a documented "local" setup turned out to read the production connection string: it applied three migrations early, and a seed script nearly wrote ten test rows to production before a row-count diff caught it. Both were harmless only because every migration was additive and nullable.

So the unattended path now accepts additive migrations only. A verify job validates the schema with fake connection strings, runs the tests, and scans each new migration file for destructive SQL. The first version blocked the one historical DROP TABLE we tried. A separate review pass the same day fed it crafted SQL, and renames, type changes, five other kinds of DROP, UPDATE ... SET and a DROP split across two lines all got through. After the fix it blocks 16 of 16 destructive cases and passes 6 of 6 additive ones. Local tooling now runs through a wrapper that refuses to start if the environment points at the managed database.

A release gate with counts

Every merge to production now goes through a fourteen-line checklist, G1 to G14, carried in the pull request template, and a line is ticked only with a link to its evidence. The data lines are the ones that carry weight:

• A restored backup. Before the first gated release we restored a full dump into a throwaway Postgres and matched exact row counts on every table.
• Counts with a verdict. A script snapshots per-table counts before and after and fails if a protected table shrank. It runs one SELECT inside a read-only transaction, and we tested it against eight failure scenarios first.
• A rehearsal. Migrations run on a branch of the production database first; for v3.4, three migrations took 4 seconds there with counts equal and no drift.
• Checksums. On September 30 two migrations were applied with md5 checksums over every existing column of the protected tables, before and after: identical.
• A rollback point. An annotated tag on the commit production is running, set before the merge.

On October 7, 2026 we also verified the database's point-in-time restore: a test branch taken from a timestamp before an edit reproduced the row as it had been.

Results

The runtime figures are three nights apart: the September 3 run never finished (the restart above) and the September 4 run failed on a database pooler outage. Most of the drop came from the breaker.

Related reading

• Silent zeros, four times a pipeline said success and was not