Alerts that can never fire: judge by the evaluator, not the preview

Most alerting reviews count things: policies, conditions, thresholds, enabled flags. The counts can look healthy while some conditions are unable to open an incident and others open incidents that reach no one.

This article is about conditions that are saved, look right in the editor, and produce nothing. The examples use New Relic, but streaming evaluation, aggregation windows, gap filling and routing filters exist in every modern alerting tool. The worked case is a client's order-to-cash policy that we audited and repaired in September 2026.

Three ways an alert fails

1. It watches nothing. The query returns no data, so there is nothing to breach.
2. It can never be satisfied. Data flows, but the shape of the signal can never meet the term.
3. It reaches no one. It fires, and no workflow, channel or person receives it.

Counting policies tests none of these. Each needs its own check.

The preview is a backtest

The chart in a condition editor answers "would this have fired against history?". It runs your query through the query engine and draws alert markers on the result. It does not run the evaluator. New Relic says so in small print under the chart ("This chart is not using streaming data"), and the chart will show critical events for a condition that has never evaluated a window.

Three properties of the real evaluator make the preview more misleading still:

• Evaluation runs forward only, and a save restarts it. In the case we measured, a condition saved at 09:15:04 evaluated its first window at 09:20:00, opened an incident at 09:26:21 and sent its email at 09:26:27, about eleven minutes from save to proof. Edit the condition three times in ten minutes and you have tested nothing.
• Close times can be backdated. We saw an incident whose closeTime read 11:15:00 while its close row was written at 12:07:59. Read the row's own timestamp to learn when it closed.
• Condition and policy ids are different numbers. Filtering incidents by the policy id returns nothing, which looks like "the alert did not fire".

On the client policy, a repair made in early September had been recorded as "confirmed firing" on the strength of the chart. When we checked the evaluator's records on September 18, it had changed the query, left every signal setting as it was, and opened no incident.

Walk the chain with the evaluator's records

Every stage writes its own event, and each answers one question. The five queries, in chain order:

-- 1. Did the condition evaluate at all? One row per evaluated window.
SELECT * FROM NrAiSignal WHERE conditionId = <condition-id> SINCE 1 hour ago
-- 2. Did a window breach and open an incident?
SELECT * FROM NrAiIncident WHERE conditionId = <condition-id> SINCE 1 hour ago
-- 3. Did the incident become an issue?
SELECT * FROM NrAiIssue SINCE 1 hour ago
-- 4. Did a workflow send it, and did the destination accept it?
SELECT timestamp, status, workflowName FROM NrAiNotification SINCE 1 day ago
-- 5. Why a delivery failed (HTTP codes live here, not in step 4)
SELECT * FROM NrIntegrationError SINCE 1 day ago

The first empty step is where the problem lives. An empty step 1 means the condition is not evaluating: wrong query, no data, or a facet that excludes it. Rows in step 1 with none in step 2 mean the shape cannot satisfy the term. Rows in step 3 with none in step 4 mean routing. Step 5 matters because a webhook refused with a 401 showed up in our case as a failed notification with an empty error field, and the HTTP code appeared only in the integration-error stream.

Why a signal can never satisfy its term

The instinct is to move the threshold. In our experience the cause is more often the signal's identity, the clock that evaluates it, or the state it is already in.

The window is shorter than the data's cadence, and gaps are filled with zero. All six conditions in the client's policy had this defect. Each used a one-minute window, event-flow aggregation, empty windows filled with 0, and a term of "above 0 for 5 minutes, in all windows". The data arrived once every five minutes, or in one batch each weekday evening. Four of every five windows were empty, became zero, and broke the run of five non-zero windows the term needed. Any incident that did open was closed by the next empty window. In 30 days the policy opened no genuine incident.

Event-flow aggregation on sparse data. Event flow closes a window only when a later event arrives. A burst followed by quiet leaves its window open, and it is never evaluated. Sparse or bursty data needs an event timer, which closes windows on the wall clock.

The term is impossible or inverted. Read each term aloud against its query. A count BELOW 0 can never be true. "No orders shipped" written as count ABOVE 0 fires on every good day.

Absence needs loss of signal. When data stops, evaluation stops, so a threshold on "nothing arrived" never trips. An absence check uses loss of signal, with the threshold term as a placeholder.

The failure is the baseline. Measure the failure rate before alerting on "any failure". In the client's policy, one flow fails most of its runs by design. "Failures above 0" on that flow would open one incident that never closes. Alert on a rate above the measured ceiling, or on the absence of success for longer than the longest normal gap.

The query changed but the settings did not. Swapping an event query for a gauge, or for any source with a different cadence, leaves the window, aggregation, gap fill and duration behind. Re-derive all of them from the new data.

The facet splits the signal. Each facet value is its own signal. One business-error feed put a per-run id in the attribute we faceted on, so every event created a fresh one-point signal that never accumulated. If the grouping attribute is unstable or often null, drop the facet until the data is fixed.

Repairing the six

We measured 7 to 30 days of each signal first, then applied new shapes over NerdGraph from a script kept in git, so the change is reviewable and repeatable. A sketch of that approach, with one condition shown:

# Window matched to a 5-minute cadence, closed on the wall clock, no zero fill.
SIGNAL = ("signal: { aggregationMethod: EVENT_TIMER, aggregationWindow: 300, "
          "aggregationTimer: 300, fillOption: NONE }")
# A term is mandatory; for absence checks it is a placeholder that can never be true.
PLACEHOLDER = ("terms: [{ operator: BELOW, threshold: 0, priority: CRITICAL, "
               "thresholdDuration: 300, thresholdOccurrences: ALL }]")

def lost_after(seconds):
    return (f"expiration: {{ expirationDuration: {seconds}, "
            "openViolationOnExpiration: true, closeViolationsOnExpiration: false }")

conditions = {
    "<condition-id>": {
        "name": "Orders: none received in 4h",
        "query": "SELECT count(*) FROM OrderReceived",
        "shape": f"{SIGNAL} {PLACEHOLDER} {lost_after(4 * 3600)}",
        "description": "No orders for 4 hours (normal gaps reach 3h). Check the sync job and ingest.",
    },
}

for cid, c in conditions.items():
    body = (f'name: {json.dumps(c["name"])} description: {json.dumps(c["description"])} '
            f'nrql: {{ query: {json.dumps(c["query"])} }} {c["shape"]}')
    result = nerdgraph(f'mutation {{ alertsNrqlConditionStaticUpdate(accountId: <account-id>, '
                       f'id: "{cid}", condition: {{ {body} }}) {{ id }} }}')
    print(cid, result.get("errors") or "OK")

The condition on current state ("orders waiting for correction") moved to a gauge read with latest(), a five-minute event timer, gaps filled with the last value, and a 30-minute loss of signal. Last-value fill alone is risky: if the feed dies, the last value freezes and looks healthy. Paired with loss of signal, a dead pipeline raises an alert.

The flow that fails most runs by design got the condition "no successful acknowledgement in 4 hours". Every description now reads as "what it means, what to do", because it is the text that lands in the email. By our records, the repaired conditions have fired on real failures since September 2026.

Sparse events: one alert per event is not how alerting works

A team asked for one email per business validation error. We tested it rather than argue. Four events, three minutes apart, under an event timer with one-minute windows, as the evaluator recorded them:

NrAiSignal:       11:05=1  11:06=0  11:07=0  11:08=1 ... 11:14=1   then no windows at all
NrAiIncident:     one incident, opened 11:05, still open 50 minutes later
NrAiNotification: one email

Alerting is state-based. While an incident is open, no new one opens. Once the stream went quiet, no further windows were evaluated, so the recovery period never elapsed, and the incident time limit was the only thing that could close it. The shape that works for sparse, bursty events:

nrql:   SELECT count(*) FROM <BusinessErrorEvent>
signal: EVENT_TIMER, aggregationWindow 60s, aggregationTimer 60s, fillOption STATIC 0
terms:  ABOVE 0, AT_LEAST_ONCE, thresholdDuration 60
limit:  violationTimeLimitSeconds 3600      # what closes a sparse incident
workflow notificationTriggers: [ACTIVATED]  # adding CLOSED doubles every notification

We cut the limit from three days to one hour, and the team accepted up to an hour between emails. If one ticket per error is the real requirement, a scheduled query that opens tickets fits better than an alert condition.

Watching nothing and reaching no one

The other two failures are cheaper to find. For each condition, run its query as written over a long window, and check that the metric or event exists at all:

-- Does the data a condition watches exist?
SELECT count(*) FROM Metric WHERE metricName LIKE 'apm.%' SINCE 30 days ago

Zero rows is a finding, and a condition on that data will read as healthy indefinitely.

For routing, list every policy id and check that some workflow filter includes it. Workflows are filters: every matching workflow fires, there is no "else", and an issue no filter matches goes nowhere. Add a catch-all workflow, and build alerts so that the policy, the condition and the workflow filtered on that policy's id are created together.

One audit trap applies to all of this: a query that fails on a syntax error returns nothing, and nothing can be misread as "none found". Check the response for errors before recording an empty result.

We used this in Self-service observability for a consumer-health company.