Running a three-region clinical platform on self-hosted Swarm

The situation

The platform is four web applications. Each runs as several containers (nginx, PHP, a background worker) on Docker Swarm clusters on Azure, with one production cluster per region and shared staging and UAT. The clusters sit inside a compliance boundary because they handle patient data, and they are hosted on a managed platform where some ordinary operations, such as adding your own SSH key, are not available.

The client's own developers build the apps, and we were brought in for the infrastructure side.

When we arrived, a hosted CI runner built the images, pushed them to a private registry, and then sent an HTTP POST to a webhook per service. There were about a dozen webhooks across the four apps. A POST succeeds when the platform accepts the request. It says nothing about whether the service took the new image, so the CI job was green either way.

What we did

Put the deploy inside each region

Calling into the swarm from CI would have meant opening production to inbound connections from CI's address ranges. We installed a self-hosted GitHub Actions runner in each region instead, inside the cluster network, and we maintain them. The runner polls GitHub over outbound HTTPS, so nothing calls into production. The deploy job runs on it: log in to the registry, pull, update each service, prune, log out. If Swarm rejects the update, the command returns an error and the job fails.

A simplified version of the deploy job, written for this page:

  deploy:
    needs: build
    runs-on: [self-hosted, "${{ vars.DEPLOY_RUNNER_LABEL }}"]
    environment: prod
    steps:
      - name: Update services
        run: |
          echo "$REGISTRY_TOKEN" | docker login registry.example.com -u ci --password-stdin
          for svc in nginx php worker; do
            image="registry.example.com/app/${svc}:${GITHUB_SHA}"
            docker pull "$image"
            docker service update -d --with-registry-auth \
              --image "$image" "app-${svc}"
          done
          docker image prune -f
          docker logout registry.example.com

--with-registry-auth hands the swarm the registry credentials. Without it, a swarm pulling from a private registry cannot resolve the tag to an image and nodes can keep running the image they already hold.

Mapped twelve targets that were not the same

Four apps in three regions make twelve production targets, and they did not share a naming scheme. A single templated job would have pointed at services that do not exist.

We inventoried the real service names per cluster first, then opened one pull request per app per region, proving the path on one app in one region before rolling out. In three days in January 2025, every app deployed the same way in every region. Two unrelated CI breakages were cleared on the way: a PHP version pin for Composer in one build, and a temporary workaround for a third-party package repository's signing key, reverted within the hour.

Later the client's developer replaced our per-branch runner label with a repository variable and moved the registry login to the standard Docker login action. Both are improvements, and both are their work.

Opened a way into machines we could not SSH into

Operating the fleet needs a shell as the platform user, and on the managed VMs you cannot SSH in to add a key, because you need a key to SSH in. We used Azure's control plane, which can run a one-off command on a VM without any network path to it, to add the key. The team now has a way in that does not depend on SSH.

Rehearsed recovery on a clone

In June 2025 we rehearsed bringing a region's production swarm back on restored Azure VMs, and the drill is now rehearsed once a year. A clone of production gets almost every default wrong: new private IPs, a manager that still believes in the old cluster membership, placement by node labels that no longer match, TLS that asks a public certificate authority, and queues that would process real jobs and send real email.

The runbook walks through six steps: force a new single-manager cluster and re-join the workers; rebuild placement labels from a hostname map and force-update every service; add test hostnames to the Traefik routers; switch Traefik to TLS that needs no public certificate authority; scale Redis to zero so queued work stops; and deny all outbound traffic from the recovery network once internal traffic is allowed. The label step, as an illustration:

# restored nodes keep their hostnames but lose their labels: rebuild, then re-place
for node in $(docker node ls -q); do
  host=$(docker node inspect "$node" --format '{{.Description.Hostname}}')
  case "$host" in
    app-node-1) docker node update --label-add role=web "$node" ;;
    db-node-1)  docker node update --label-add role=db  "$node" ;;
    *) echo "no label rule for $host" ;;
  esac
done
for svc in $(docker service ls -q); do docker service update --force "$svc"; sleep 5; done

Checked a CVE against configuration

In July 2026 an nginx CVE arrived whose exposure depended on configuration, so a version check could not answer it. We reviewed every virtual host config on the platform for the pattern that made it reachable. None used it, and the platform was not exposed.

Results

• 12 production targets (4 apps, 3 regions) moved from webhooks to in-cluster runners in three days, through 20 pull requests, 19 merged.
• About a dozen webhook calls replaced by deploys that fail when the update is rejected.
• All but one production deploy green from August 2025 to September 2026 (none failed, one cancelled). This measures the pipeline after the change; the CI history from before August 2025 has expired, so there is no before figure to compare.
• A recovery runbook for a production swarm on restored VMs, rehearsed once a year, and a privileged access path that does not depend on SSH.
• Every nginx virtual host config reviewed against a configuration-dependent CVE, none exposed.

Related reading

• Pin the digest: why Swarm ran the previous build