Three years running a manufacturer's hybrid estate

The situation

The client builds industrial equipment and runs a connected platform on AWS that lets its customers follow how their machines are doing. Around it sits a wider estate of Windows and Linux servers in on-premises virtualization, AWS and Azure. Most of it had been built by other people, and the hosting work had changed hands before we arrived.

A legacy Zabbix installation watched the estate and opened a ServiceNow incident for every problem it detected. The plan was to move monitoring to New Relic, which the platform team already used for application alerting. Taken literally, "migrate the monitoring" meant carrying every check across and delivering the same ticket volume on a newer screen.

We worked inside the client's process for three years. Work arrived through ServiceNow queues with task SLAs, procedures lived in the client's wiki, and production access went through single sign-on and session managers rather than SSH keys.

What we did

Counted what the alerts led to

Before designing anything we exported 30 days of monitoring-raised incidents and counted what had happened to each one. The numbers set the brief:

• About 650 tickets, a little over 20 a day.
• About 95% of them closed themselves when the condition cleared. None was resolved at first line.
• Nearly all of those raised as critical closed themselves.
• The median ticket stayed open for about four and a half hours, long after its condition had gone.

Some closures read "ignore this type of alert according to the SOP". The service desk had already learned which checks to discard, by hand.

The count takes a few lines against any ITSM export. A version of it, written for this page:

# share of monitoring tickets that closed themselves, and the noisiest hosts
import csv, collections

rows = list(csv.DictReader(open("monitoring-incidents-30d.csv")))
auto = [r for r in rows if "auto-resolved" in r["resolution_notes"].lower()]
print(f"{len(rows)} incidents, {len(auto)} self-resolved "
      f"({100 * len(auto) / len(rows):.0f}%)")

by_host = collections.Counter(r["host"] for r in rows)
for host, n in by_host.most_common(10):
    print(f"{n:5d}  {host}")

Built a kill list for the noisy checks

A ticket that closes itself 95% of the time points at the wrong alert design: the duration window is shorter than the signal, or the state belongs on a dashboard. We built a kill list from the export. Checks that the procedures already said to ignore, and conditions that routinely cleared before anyone could act, went on the list and were not recreated in New Relic.

Took the monitoring server out of its own queue

By host, the largest single source of tickets was the monitoring server itself: about one in eight. Its sender processes, its proxies' clocks and agents that briefly stopped answering all landed in the same queue as real outages.

We set up out-of-band monitoring of the monitoring, with its own health check and owner, outside the service desk. For "agent unavailable" tickets we wrote a short triage, run on the agent host, that tells a path problem from a host problem in a couple of minutes:

# watch the agent port and read the direction of the traffic
sudo tcpdump -i any -n tcp port 10050
# nothing arrives          -> routing, or the wrong address in the server's host entry
# arrives, nothing leaves  -> the host firewall is dropping the reply
# both directions          -> look at the server side next

Moved the hosts with their routing

The new platform had to cover the same hosts before the old tool could go. We built a starting list of several hundred hosts across the on-premises, Azure and AWS estate. For the Linux side we exported each host's monitoring groups (account, region, application, environment), because those groups were how alerts were routed and owned, and carried them over as tags. A host on the list with no data in the new platform showed up as a gap before the cutover. The legacy tool was retired in 2022.

Ran the estate by runbook, Terraform and Ansible

The rest of the three years was operations on a platform we had not built. In the first month we inventoried the Linux estate across every environment, with a column for who owned each machine. Recurring failures (disks filling with logs, backup snapshots never reclaimed, application servers reporting the wrong state) each got a runbook written during the incident, with the safe default built in: keep the newest copy, never delete the last one, ship destructive steps disabled. Fleet changes such as central log forwarding went through idempotent Ansible.

On a shared Terraform stack owned by other teams, we added a set of hostnames, the single sign-on redirect URIs and a DNS-validated certificate in one change, applied in dependency order with targeted applies so nothing else on the stack moved.

In the later part of the engagement we also led the client's ServiceNow team alongside the engineering work.

Results

• Baseline (measured): about 650 monitoring tickets in 30 days, 95% self-resolving, none resolved at first line, about one in eight raised by the monitoring server about itself.
• After the move (by our records): redundant alerting down by about 80%, with the kill list in place and the monitoring watched out of band. The legacy tool was retired in 2022.
• Coverage: several hundred hosts on the onboarding list, Linux hosts carrying their routing groups as tags.
• Operations: three years of runbooks, ordered Terraform changes on shared stacks and Ansible fleet changes, and the client's ServiceNow team led in the later part of the engagement.

Related reading

• An engineering simulation cluster on Azure scale sets, a separate stream of work in the same engagement.