Observability for Docker on VMs.
Alerts that fire, and reach someone.

Most of the monitoring we are asked to look at has the same three faults, and none of them shows on a dashboard.

• Alerts that can never fire. A condition queries a metric the account does not collect, so it stays green forever. In our own observability account, 3 of 7 enabled conditions were watching data that had never existed.
• Alerts that reach no one. The condition fires, but its policy is in no notification workflow, so nobody is paged. Our one real CPU alarm had fired as critical and gone nowhere.
• Alerts that close themselves. At an industrial manufacturer we counted 30 days of monitoring tickets: about 650, and 95% closed on their own before anyone acted. The service desk had already learned which ones to ignore.

Each one ends the same way: the team hears about an outage from a customer, or stops reading alerts at all.

Measure before we change anything

We start from your own history: an export of the last month of alerts or incidents, counted by what happened to each one, which hosts raise the most, and how much of the noise is the monitoring reporting on itself. The count sets the brief.

Prove each alert can fire, and that someone gets it

For every condition we ask two questions: does its query return data, and does a notification route include it? Then we test the check against a real fault and confirm the message arrives. On one estate this found four health checks that could not detect the failure they were written for; one always reported the database as up. All four were fixed before any of them hid an outage.

One alert per outage, and one when it recovers

We latch each alert so the first failure pages, later failures stay quiet, and recovery is announced. Checks that close themselves most of the time go on a kill list and are not carried forward.

Alerting as reviewable code

We manage alert conditions, policies, notification routes and uptime checks in Terraform, built as one unit so a policy always has a delivery path.

Watch the monitoring from outside

The monitoring server is often one of its own largest sources of tickets, so we give it an out-of-band health check and an owner.

A migration is the cheapest moment to drop bad alerts. When we moved an industrial manufacturer's monitoring from Zabbix to New Relic, we built the starting list of several hundred hosts across on-premises, Azure and AWS, carried each Linux host's monitoring groups across as tags so alerts kept their routing and owners, and flagged any host on the list with no data before the cutover. Checks on the kill list were not recreated. Redundant alerting fell by about 80%, by our records, and the legacy tool was retired in 2022.

Most engagements start with the Production Reliability Check: a read-only, fixed-price review of your alerting and your release path, with a written report and a 90-day plan. If you would rather talk first, book a 30-minute discovery call and walk us through your stack. For the release side, see deploy safety.