How we run our own infrastructure as code

The situation

Our estate is small: four Terraform stacks (the cloud account, DNS, the hypervisor and the observability account) plus configuration management for the hosts, in one repository, run by one engineer. Most of it had been code for years. By the summer of 2026 that was not the same as being run as code. Each stack had its own CI workflow, written to the same shape on purpose, and our monitoring dashboard was green.

Two questions shaped the work. Can a merge change infrastructure? And does an alert that looks healthy reach anyone? Both answers turned out to be different from what we believed.

What we did

A merge that applied infrastructure

On September 22, 2026 a pull request added one comment to a hypervisor module. Three minutes after the merge, a pending change ran two configuration playbooks. Both playbooks are idempotent and the run ended with failed=0, so nothing broke. But the rule we believed, that merging never applies, was false: one of the three per-stack workflows applied on push to main. We had read all three files that day. Compared line by line, they differed in six ways.

One pipeline: plan on the pull request, apply by dispatch

We replaced the four workflows (about 1,240 lines of near-duplicate YAML) with one caller and one reusable stack workflow (296 and 459 lines), so the plan and apply logic exists once. The caller lists each stack as a literal job, which keeps the dependency graph visible in the Actions UI.

• Pull requests plan every stack with read-only cloud credentials and post one comment per stack.
• A push to main plans again, uploads each plan as an artifact, and stops.
• Applying is a manual dispatch. An apply never applies a fresh plan; it applies the reviewed artifact built on main. The dispatch finds the newest unexpired plan artifact from main, downloads it and applies that file. If state has moved since the plan was made, Terraform refuses the saved plan, and we treat that refusal as intended.

Cloud access uses OIDC with two roles: a read-only one that pull requests can assume for plans, and a write role that trusts main only. A dispatch with the apply box unticked doubles as a drift survey: it plans every stack and fails if any of the four would change.

A destroy gate on the plan that gets applied

Any plan that deletes or replaces a resource needs that stack's own checkbox ticked on the dispatch. A replace counts, because a replaced VM is a rebuilt VM. The first version took a typed list of stack names, and a typo (aws-platform for aws/platform) read as "no permission" and skipped the stack. A checkbox cannot be mistyped.

We tested that first version against a plan that really deleted things before relying on it: 23 unused cloud resources we had adopted into Terraform in order to remove them. Without approval, the stack failed with zero destructions while the other stacks applied normally. With the box ticked, all 23 were destroyed, and we checked in the console that everything meant to survive had survived.

Writing the companion article, we found that the gate counted destroys in the fresh plan the apply job makes to decide whether there is anything to do, and then applied the downloaded artifact. A reviewed plan with a destroy could pass on a day when the fresh plan happened to be clean. Since October 13, 2026 the gate runs after the download, on the same file the apply step uses. An illustration:

# After the reviewed artifact is downloaded, before it is applied.
terraform show -json tfplan > reviewed.json
count() { jq "[.resource_changes[]? | select($1)] | length" reviewed.json; }
del=$(count '.change.actions == ["delete"]')
rep=$(count '.change.actions == ["delete","create"] or .change.actions == ["create","delete"]')

if { [ "$del" != "0" ] || [ "$rep" != "0" ]; } && [ "$CONFIRMED" != "true" ]; then
  echo "::error::$STACK reviewed plan destroys or replaces. Tick 'Allow DESTROYS in $STACK' to apply."
  exit 1
fi
terraform apply -input=false tfplan     # the same file the gate just read

The new gate passed on the reviewed plan in its first two real applies the same day.

An alerting audit

The first inventory of our observability account read 4 policies, 7 enabled conditions and 3 uptime checks, and looked healthy. The same morning, September 23, 2026, a host CPU alert had fired. So we ran a second pass that asked two questions of everything: does each condition's query return data, and does any notification workflow include each policy?

• Three conditions queried application-performance metrics. A 30-day count of any such metric returned 0, and the account had no instrumented applications at all. Those conditions could never fire.
• The CPU condition was real and had fired CRITICAL. Its policy was in none of the workflows' filters, so it reached no one.

We first brought the three working uptime checks under Terraform with a module that builds the monitor, policy, condition and workflow as one unit, so a policy always has a delivery path. Our rule for adoption is that the plan must read zero changes. The first plan showed two (a different aggregation window on one check, a different name on one condition), and both became explicit inputs; the final plan imported 12 resources with 0 added, 0 changed and 0 destroyed.

On October 13, 2026 we finished the fix. We deleted the policy that held the three dead conditions and the unrouted CPU condition, rebuilt the CPU alert through a module, routed it to the chat channel we read, and sent a test notification, which was delivered. One detail cost us an apply: New Relic binds a notification channel to a single workflow, so reusing an existing channel fails with INVALID_PARAMETER. The module now creates its own channel on a shared destination.

Results

Related reading

• A destroy gate for Terraform on GitHub's free plan
• Alerts that can never fire: judge by the evaluator, not the preview