Self-service observability for a consumer-health company

The situation

The company was consolidating its monitoring onto one New Relic account. Azure resources, virtual machines and container apps were the easy part, since standard agents and integrations cover them.

The business ran on SaaS products as well: a data-integration platform, a supply-chain planning system, a marketplace order hub and an ITSM tool. None had a native integration. A team that wanted to know whether its nightly feeds ran had no good option.

An ingestion repository existed when we joined, with a minimal bootstrap and a first few integrations. Working inside the client's platform team, we built the self-service platform on it, then the integrations, the alert repairs, the ITSM routing and the dashboards.

All work ran in a managed virtual desktop behind a corporate proxy, against production systems owned by other teams with their own change calendars.

What we did

Rules before data

Before any data flowed we agreed the ground rules with the platform lead. Every Terraform-managed resource carries five tags (environment, service, owner, cost center, tier), and Azure Policy enforces the contract. Monitoring follows tags, never pasted entity IDs. Ingest keys and control-plane keys stay apart. Baseline alerts live as Terraform in a governance repository, custom alerts go through a request with a named approver, and New Relic holds the alert state, with the ITSM tool following it.

A platform teams can use without seeing a key

A team adds one entry to a Terraform map and one code folder. Each entry becomes its own resource group, Key Vault, storage account, Function App and managed identity. Variables hold secret names only, rendered as Key Vault references, so no secret value reaches git, variables or state. The platform's ingest key sits in a separate vault that no team identity can read.

A simplified version of the pattern, with placeholder names:

# Teams name their secrets; only references reach app settings and state.
integrations = {
  orders-feed = { secrets = { "orders-api-key" = "ORDERS_API_KEY" } }
}

app_settings = merge(
  { "INGEST_KEY" = "@Microsoft.KeyVault(SecretUri=${var.platform_vault_uri}secrets/ingest-key/)" },
  { for name, setting in each.value.secrets :
      setting => "@Microsoft.KeyVault(SecretUri=${module.team_vault[each.key].uri}secrets/${name}/)" },
)

A second tenant of an existing integration is a second deployment entry pointing at the same code, and every record carries its source environment so production dashboards never count test rows. Code deploys only after a successful infrastructure deploy, and it checks out the commit that triggered it.

Pollers that prefer a duplicate to a gap

Every integration follows one poller template. It advances its checkpoint only after a successful send, because a duplicate shows up in a query and a gap does not. It checks response shape and content type rather than trusting the HTTP status, since one vendor answered a bad session with its sign-on page and a 200.

The supply-chain planning system was mapped through a probe harness that refuses, before sending, any request not on a read allowlist, because the same API can delete planning data. The monitoring model built from that sweep exposed a production bug: the feed's row cap defaulted to the API's page size, and one table was larger than a page, so rows were dropped every cycle. We separated the page size from the ceiling and deployed the fix.

An ITSM integration whose token lives 30 minutes

The client required OAuth for alerts flowing into ServiceNow, and New Relic's webhook destination stores a static header. We built a relay on the platform: an Azure Function that mints a token and writes it into the destination. The design document implied 100 days. We measured the access token at 1,799 seconds; the 100 days belonged to the refresh token. The planned weekly schedule would have left routing dead for most of every week, so the relay runs every 20 minutes and records an audit event on every run.

The first live delivery failed with a 401. We first advised dropping OAuth for an API key, then withdrew that the same day when one token got a 200 from two table endpoints and a 401 only from the integration's endpoint. The cause was one access policy, which the ServiceNow team changed. We then agreed the incident payload with them over nine live test issues, since every incident had been landing at priority 5 with raw JSON as its description. The method is in the article linked below.

Alerts that could not fire

The order-to-cash policy behind a business-flow view had six conditions, and all six shared one defect: one-minute windows, empty windows filled with zero, and a five-minute "all windows" term against data that arrives every five minutes or once a day. None opened a genuine incident in 30 days. A repair made earlier in September had been recorded as confirmed on the strength of the preview chart.

We measured 7 to 30 days of each signal, then reshaped all six over NerdGraph from a script in git: windows matched to each feed's cadence, current state read from a gauge, and loss-of-signal paired with any last-value fill so a dead feed raises an alert. Where failure is the baseline we alert on missing success. One flow fails most of its runs by design, so its condition became "no successful acknowledgement in 4 hours".

Results

The relay ran 1,284 times between September 3 and 21, 2026, every run healthy, with no failed destination write. In the seven days to September 29, 34 of 34 deliveries to the development instance succeeded, and incident priority now follows alert priority. By our records, the rebuilt order-to-cash alerts have fired on real failures since September 2026. The planning feed reads its tables whole.

We also built nine production dashboards and two business-flow views. One query on the order dashboard answered a question the business had no number for: how long a marketplace order takes to become an ERP sales order.

Related reading

• Keeping an ITSM integration alive when its token lives 30 minutes
• Alerts that can never fire: judge by the evaluator, not the preview