Case study · A North American medical-device company

Infrastructure for a medical-device company's clinical platform

Since December 2024 we have run the infrastructure under a medical-device company's clinical platform, and today no deploy opens SSH from CI to any of its hosts.

4 min read

At a glance

Sector
A North American medical-device company
Duration
since December 2024
Our role
Infrastructure engineer; the client's in-house developers do the application work
Team
TQM Invest, one infrastructure engineer, with the client's developers
  • AWS EC2
  • VPC
  • RDS
  • Docker Swarm
  • Docker Compose
  • nginx
  • Cloudflare
  • GitHub Actions
  • PostgreSQL
  • MariaDB
  • Redis
  • Ansible
  • Bash

Headline results

  • No deploy opens SSH from CI: host addresses, ports and private keys left the CI secrets in January 2026.
  • Postgres moved from 14 to 17 on every clinical environment, with every image that runs pg_dump brought to the matching client.
  • A research database refresh went from a hand-run script with five "press Enter" prompts to a nightly job that refuses to load unless the client's listed PHI columns are gone.
  • A production fault that brought up the previous build was diagnosed and fixed in August 2026, confirmed against a live deploy.

The situation

The company runs a clinical web application and a few supporting services for patients and clinicians, all on Docker hosts in AWS, with each jurisdiction's patient data kept in that jurisdiction. We joined with a request for quick wins. The client's developers own the application and its release flow; we own the hosts, the network and the deploy path.

What we inherited worked, with the habits of a team that had moved fast. CI deployed by copying files over scp and running a script over ssh, with host-key checking off. Some containers had been started by hand. Every release recreated every container. Each host requested its own TLS certificate at deploy time.

What we did

A network laid out for health data

Between December 2024 and March 2025 we re-laid the AWS network under the platform: a new VPC and subnet layout, with the environments segmented to support the client's HIPAA obligations. UAT moved first. By our records, production moved in one afternoon in March 2025, driven by a checklist we wrote in advance and kept as the runbook for later moves. A static public IP followed, so the next change would not mean re-pointing every deploy.

Releases that roll

We read the shell history on each host, turned the hand-typed docker run commands into an idempotent bootstrap script and moved the stack onto Docker Swarm. Deploys became one rolling update per service. Database migrations moved into a single worker process, so they run once.

We then took certificates off the hosts: every application host now sits behind the Cloudflare proxy with an origin certificate, mounted read-only. A deploy no longer needs port 80 free or a certificate authority to answer.

A second jurisdiction, and no SSH from CI

In mid-2025 we stood up the second jurisdiction's production and UAT as a parallel stack, each with its own host, runner, configuration and hostname. Later that year a release renamed the first jurisdiction's hostnames so both follow one pattern.

In January 2026 we split every pipeline in two. The build runs on a GitHub-hosted runner and pushes images. The deploy runs on a self-hosted runner that lives on the target host and dials out to GitHub. Every deploy target across three repositories moved in about a week.

The deploy step on each host, in outline:

deploy:
  needs: build
  runs-on: [self-hosted, app-prod-host]   # a per-host label, never bare "self-hosted"
  environment: production
  steps:
    - uses: actions/checkout@v4
    - name: Deploy from this checkout
      run: bash .docker/deploy.sh

Production ran the previous build

In August 2026 production came up on the previous build on two deploys running, with CI green both times. Images carried one tag per environment, overwritten on every build, and Swarm can restart a service onto the image a node already holds under that tag. A second defect sat beside it: the production deploy script still read its configuration from a host directory the old SSH copy used to fill.

We diagnosed both and wrote the fix: hand Swarm the image digest read after the pull, and deploy from the runner's fresh checkout. The client's developer deployed it to production the same day, and a live deploy confirmed the new build. The method is in Pin the digest: why Swarm ran the previous build.

Data tier upgrades and a read-only fleet audit

In March 2026 we ran a read-only audit of every Docker host with Ansible: one play gathers versions with read commands only, a second renders the report on the control node. Components run as rows and hosts as columns, so version drift shows on one line. Each end-of-life component got an upgrade path.

We moved Postgres from 14 to 17 on staging and both UAT environments with a dump-and-restore runbook, and the client's developer made the production image change. Redis moved from 6 to 8, and MariaDB was upgraded. When a production server failed a status check, our read-only forensics playbook found CPU-credit throttling, and the instance's resources were then set to run correctly.

A nightly PHI-safe research refresh

The client keeps a research database per jurisdiction: a copy of clinical data with patient information removed. Refreshing it meant a hand-run script with five prompts that carried on after errors. In September 2026 we rebuilt it as a nightly job on each jurisdiction's production server, loading that jurisdiction's research server. Here "PHI-safe" means the client's listed PHI columns are proven absent before anything loads. The scrub list is the client's, kept byte for byte. Before loading, the job checks the schema for any column on that list, any column whose name looks like contact or identity data, and any table that should have been dropped. Any hit stops the run.

On its first production night it refused to load, naming six columns that only looked like PHI. Nothing reached the research database. We narrowed the check by column type, ran it read-only against the copy the failed run left behind, and both jurisdictions then passed with zero row-count mismatches. Failure emails withhold error text from the steps that touch unscrubbed data, because Postgres errors can quote row values.

Results

  • Releases roll through replicas, and certificates no longer depend on the deploy.
  • Every deploy target deploys from a runner on its own host; CI holds no host address, port or SSH key.
  • Production runs the build CI pushed, confirmed against a live deploy in August 2026.
  • The data tier is current: Postgres 17, Redis 8 and an upgraded MariaDB.
  • The research refresh runs nightly in each jurisdiction, with a schema check that has already refused one bad load.

Go deeper

Standalone write-ups of the methods in this study.

Related case studies

Running into something like this?

Thirty minutes, no commitment. Bring the problem; we will tell you what we would check first and whether we are the right people for it.

All case studies