Silent zeros: four times a pipeline said success and was not

Below are four we have met, in our own systems and on a client engagement, followed by the pattern that catches them, the code and the traps, which apply to any scheduled job.

Four cases

1. The summary that counted one kind of failure

A nightly lead pipeline of ours ran eight scrapers and printed a summary. A scraper that raised an exception counted as failed. A scraper that caught its own failure and described it in a structured error list counted as healthy. For about a week, two sources (one timing out, one rejecting its input) returned nothing every night, and every night the summary read "8 scrapers, 0 failed", exited 0 and posted a green line to chat.

Before that, the same pipeline had produced three empty runs in a row, and each fix named one more specific cause. Then a fourth appeared: an ingest step fetched 5 items, mapped 0, and recorded no error. The function that dropped the items had three reasons to drop one, and the code explaining empty results re-derived only one of them. Each new cause needed another fix at the call site.

2. The paid job that was green for nine months

Two of the scraping jobs we publish on a marketplace had never worked. One read a feed that the site renders in the browser, so the HTML it fetched held no postings. The other used an API the platform had closed to unauthenticated clients. Both exited SUCCEEDED with an empty dataset on every run for about nine months. For a buyer, an empty green run looks the same as a broken product, and our own dashboards showed green too.

3. The collector that captured yesterday

A daily collector of ours fetched end-of-day prices from a stock exchange's website after the close. One Monday the exchange had not yet published the session when the job ran. The fetch returned HTTP 200 with a valid page: Friday's. The parser read the page's own date, Friday's, rewrote Friday's 152 rows, and recorded ok. The log for those nights:

collector: status=ok session=<friday> captured=152/90
collector: status=ok session=<friday> captured=152/90
collector: status=ok session=<tuesday> captured=161/90

A separate alarm caught it the next morning, because it did not take the job's word. It asked the trading calendar which session was due and whether a good run existed for it. In the same pass, the heartbeat said "ok" and the calendar alarm said CRITICAL:

[INFO] [heartbeat] collector run: status=ok session=<friday> captured=152/90
[CRITICAL] [missing_day] no successful collector run recorded for trading day <monday>
monitor: UNHEALTHY alerts=2 worst=CRITICAL

The fix shipped the same day. The defect had been latent for about two weeks and caused one bad run.

4. The health checks that could not fail

On an application-support engagement for a European manufacturer's financial reporting application, we wrote cron checks for the file share, the app server, the database port and the load average, which mailed the team when something went down. They were first tested by flipping each condition, which made the alert fire. When we tested them instead by breaking what each one watched, three could not do their job. A sketch of the faults, with generic names:

# BEFORE: three checks that cannot fail the way they claim
out=$(timeout 5 bash -c "cat < /dev/null > /dev/tcp/${db_host}/${db_port}")
[[ -z $out ]] && echo "db up"            # a redirect prints nothing: always "up"

ps -ef | grep -v grep | grep -q "$(cat "$pid_file")"
(( PIPESTATUS[1] == 0 )) || alert app    # [1] is `grep -v`, true while ps prints anything

load=$(uptime | awk '{print $(NF-2)}' | tr -d ,)
[[ $load < 8 ]] && echo "load fine"      # string order: "10.5" < "8"

The database check tested the output of a redirect instead of its exit status, so it reported "up" with the port closed. The process check read the exit status of the wrong stage of its pipe. The load check compared strings, so a load of 10.5 passed. A fourth fault, a misnamed latch variable, made the file-share check mail on every run. We caught all of them before any hid a real outage, and replaced them with checks that test exit statuses and numbers:

# AFTER: exit statuses and numeric comparison
timeout 5 bash -c "</dev/tcp/${db_host}/${db_port}" 2>/dev/null || alert db
kill -0 "$(cat "$pid_file" 2>/dev/null)" 2>/dev/null || alert app
read -r load _ < /proc/loadavg
awk -v l="$load" 'BEGIN { exit !(l < 8) }' || alert load

The full engagement is in Automating application support on a legacy Java estate.

The pattern

In all four, the signal that said "success" measured whether the job ran, and said nothing about whether it produced what it exists to produce. Exit codes, HTTP 200, "no exception" and "the script finished" all answer "did it run?". The fix each time was to make the job state a positive fact about its output, and to let something independent check that fact.

Five rules cover it.

1. Every empty result carries a reason, decided where the rows are dropped. Do not enumerate causes at the call site. Let the function that drops items count why, and assert one invariant at the end: if items came in and none came out, record the breakdown. Keep it somewhere durable, such as a run table, as well as in a log that rotates.

2. Name the one kind of empty that is legitimate. Jobs that return only what is new are legitimately empty on quiet days. That case gets its own reason and does not fail. Everything else does. A sketch of the function that picks the reason, in order:

def diagnose_empty(fetched, parsed, filtered, seen_skipped, blocked):
    """Return (reason, message, is_failure). Order matters; each branch was a bug once."""
    if blocked and not parsed:
        return "blocked", f"Blocked on every request ({blocked}).", True
    dropped = {k: n for k, n in filtered.items() if n}
    if dropped:                                   # test the breakdown, not `parsed`
        detail = ", ".join(f"{k}={n}" for k, n in sorted(dropped.items()))
        return "filtered_out", f"Read {parsed}, filters removed all ({detail}).", True
    if seen_skipped:                              # the one legitimate empty run
        return "all_seen", f"Nothing new: {seen_skipped} already returned.", False
    if fetched and not parsed:                    # items in, nothing usable out
        return "none_usable", f"Source returned {fetched} records, none usable.", True
    return "source_empty", "The source returned nothing.", True

Write the run statistics before you raise, because a failed run is when someone reads them.

3. A reported failure is a failure. If a component can catch its own error and describe it, count that as a failure in every summary and fail the stage. Keep "the job did not run at all" as a separate alarm so it keeps one meaning.

4. Let the data carry its own date, and compare it with the date you asked for. The collector now compares the session the page served with the session the run was for. A mismatch is its own status, stale, which exits non-zero and does not count as a capture:

if required is not None and result.session_date != required:
    return record_run(status=STATUS_STALE, error=(
        f"page served session {result.session_date}, expected {required}: "
        "the source had not published it at fetch time"))

5. Alarm from the expectation, independent of the job. The alarm that caught case 3 does not read exit codes. It asks the calendar which session is due and whether a good run exists for it, so it also catches the job never running, the host being down and the cron line being deleted. Two details make it trustworthy. The alarm and the catch-up retry share one definition of "captured", so they cannot disagree. And an unknown status maps to critical, so a schema change cannot mute it:

def has_capture(store, session_date) -> bool:
    """A failed, skipped or stale row is not a capture, and neither is no row.
    Shared by the alarm and the catch-up path so they agree on 'captured'."""
    row = store.execute(
        "SELECT COUNT(*) FROM collector_runs WHERE session_date = ? AND status IN (?, ?)",
        (session_date.isoformat(), STATUS_OK, STATUS_PARTIAL)).fetchone()
    return bool(row[0])

SEVERITY = {STATUS_OK: INFO, STATUS_SKIPPED: INFO, STATUS_PARTIAL: WARNING,
            STATUS_STALE: CRITICAL, STATUS_FAILED: CRITICAL}
severity = SEVERITY.get(status, CRITICAL)   # unknown statuses map to critical

A non-trading day records a skipped row, which is what lets the alarm tell "closed today" from "never ran".

What it changed

• The nightly pipeline's morning check became one query: a run with zero raw items, zero inserted and an empty error field is a bug. Empty nights now carry a reason such as "all items were duplicates".
• The marketplace jobs now fail an empty run with a sentence the buyer can act on. Making each job explain itself also found a latent bug in every job that adopted it: a default configuration that returned 0 rows instead of 25, and a partial block that discarded 245 fetched rows.
• The collector has captured every trading session since the fix, 44 of 44 to September 30, 2026, and the catch-up run makes no request when the session is already stored.
• The legacy health checks were each proven against a real fault before the fixed versions went live.

Traps

• Turning the check off hides the symptom. When one of our jobs kept failing on an empty filter intersection, the tempting fix was to stop failing on zero. The real fix was the input form that produced the empty intersection.
• Partial drops are normal. Alerting on ordinary attrition trains everyone to ignore the alert that matters.
• A guard's own switch must fail closed. The collector's staleness guard raises on an unrecognized mode value, because a typo there would disable the guard.
• Best-effort notification must not mute the alarm. If the chat webhook fails, the alert is already in the log and still drives the exit code.
• A success notification that never arrives looks like "nothing to report". We once found a notify step on a deploy host that could never fire, because its cron line did not load the environment holding the webhook. Treat a missing message as unknown.
• Negative checks pass on empty input. "No error string in the response" is true of an empty body and of a redirect. Assert a positive marker: HTTP 200 and a field the real response always contains.
• Test a check by breaking the thing it checks. Close the port, stop the process, load the box. Flipping the operator tests only the alerting.

We used this in A nightly batch pipeline that cannot report an empty run as success.