Pin the digest: why Swarm ran the previous build

The symptom

CI is green. The deploy log shows the new image being pulled. The service restarts. The application still runs the previous release.

We met this on a medical-device company's production platform in August 2026. Production came up one build behind on two deploys running before the pattern was clear, because no step failed. Every command in the deploy did what it was written to do.

What Swarm does with a tag

A Swarm service does not hold an image. Its spec holds a reference to one. Take a typical deploy line:

docker service update --force --image registry.example.com/app/web:production app_web

The Docker CLI tries to resolve web:production to a content digest at the registry, and writes the pinned form, web:production@sha256:..., into the service spec. Every node that runs a task for the service then pulls that digest. That is the normal path, and it is why most teams never think about this.

Two things can take you off it.

Resolution can fail. Against a private registry, resolution needs credentials. If the CLI cannot reach the registry or is not authorized, it prints a warning that the image could not be accessed to record its digest and that each node will access it independently, and stores the bare tag in the spec. Each node then decides for itself what web:production means, and a node that already holds an image under that tag can run it.

The tag never changes. If every build overwrites one tag per environment, the spec you submit on each deploy reads the same as last time. --force restarts the tasks, but nothing in the update tells Swarm the image is different. Whether the tasks come back on the new build depends on resolution and pulls going right on every node, every time.

One more thing hides the problem. The docker pull or docker compose pull in your deploy script runs on one machine. If that machine is a self-hosted runner on one Swarm node, the pull warms that node's cache and says nothing about the others.

What we found

The platform's images carried one tag each, the environment branch name, overwritten on every build. The deploy script passed --with-registry-auth and then handed Swarm that tag with a forced update, one line per service.

There was a second defect in the same script. Months earlier, deploys had moved from "copy files over SSH, then run the script" to "run the script on a self-hosted runner on the host". The fresh configuration now arrived in the runner's checkout, but the production script still began by changing into the fixed directory the old SSH copy used to fill. It was reading a stale .env and a stale compose file.

We diagnosed both and wrote the fix. The client's developer deployed it to production the same day, and a live deploy confirmed the new build. The production fix combined two changes: hand Swarm the digest, and deploy from the runner's checkout. We cannot say which of the two alone would have cured that deploy, so we describe each as its own fix.

The fix: give Swarm a digest

After the pull, the local image knows the digest it was pulled as. Read it, and give Swarm the immutable repo@sha256:... reference. There is nothing left for a node to resolve, and every node pulls the same bytes. This is the change we recommend for any Swarm deploy that reuses tags.

The simplest form, per service:

docker compose -f stack.yml pull
ref="$REGISTRY/web:$ENV_TAG"
docker service update --force --with-registry-auth \
  --image "$(docker image inspect --format '{{index .RepoDigests 0}}' "$ref")" \
  app_web

docker image inspect runs after the pull, so RepoDigests holds the digest of the image just fetched.

Making it sturdier

That one-liner has two weak points.

An empty substitution changes the command. If inspect fails (the pull failed, a name is misspelled), $(...) expands to nothing, and --image takes the next argument as its value. The deploy then does something you did not write. Resolve the digest into a variable first and stop without one.

index .RepoDigests 0 may not be your repository. An image pulled or pushed under more than one name has several entries. Pick the entry that matches the repository you deploy.

The shape we recommend, with a check after the update that the spec holds the digest:

set -euo pipefail

deploy_service() {
  local service="$1" ref="$2"            # ref: registry/repo:tag you just pulled
  local repo="${ref%:*}" digest have
  digest="$(docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$ref" \
            | grep -m1 -F "${repo}@sha256:")" || true
  if [ -z "$digest" ]; then
    echo "::error::no registry digest for $ref; refusing to hand Swarm a tag" >&2
    exit 1
  fi
  # no --detach: the CLI waits for convergence and fails if the update does not converge
  docker service update --force --with-registry-auth --image "$digest" "$service"
  have="$(docker service inspect --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}' "$service")"
  [ "$have" = "$digest" ] || { echo "::error::$service runs $have, expected $digest" >&2; exit 1; }
}

cd "$(dirname "$0")"                      # the runner's checkout is the only source
set -a; . ./.env; set +a
docker compose -f stack.yml pull
deploy_service app_web    "$REGISTRY/web:$ENV_TAG"
deploy_service app_worker "$REGISTRY/worker:$ENV_TAG"

Leaving out --detach makes each update wait until its tasks converge, or fail the step if they cannot. A slow or broken rollout then shows up as a red deploy.

Then fix the tag too

A digest stops Swarm from running the wrong build. It does not tell you which build is live. If the only tag is the branch name, the registry cannot say which commit production runs, and a rollback means a rebuild.

Tag each build with something unique, the commit SHA or a release number, alongside the environment tag, and push both. A running service's spec then reads web:<sha>@sha256:..., and "what is live" becomes a one-line question to Swarm.

The traps around it

Deploying from a copy on the host. If your deploy moved from SSH copies to a runner on the host, check that no script still changes into the old copy directory. Start deploy scripts with cd "$(dirname "$0")" and let the checkout be the only source.

export $(cat .env) mangles values with spaces. Use set -a; . ./.env; set +a.

Fixes that live on one environment branch. If each environment deploys from its own branch with its own copy of the deploy script, a fix committed to the UAT branch stays there. In our case the fix sat on non-production branches for weeks. Put the deploy script on the release path, so every environment inherits it by merge.

--no-resolve-image. This flag tells the CLI not to resolve tags at all. With reused tags it guarantees the "each node decides" path. If you find it in a script, ask why it is there.

--force without --image. It restarts tasks on the spec's current image. That is useful after a config change, and it does not deploy anything new.

Checking what runs. The spec shows what you asked for. The tasks show what you got:

docker service inspect --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}' app_web
docker service ps --no-trunc --filter desired-state=running \
  --format '{{.Node}} {{.Image}}' app_web

If the spec has no @sha256: part, Swarm is resolving a tag on your behalf.

We used this in Infrastructure for a medical-device company's clinical platform.