The deploy finished. The panel said done, the build log was clean, and the commit hash on the build matched what I had just pushed. An hour later a user sent me a screenshot of an admin page still showing the previous release's links. Nothing had failed. The new image existed, but it was on the wrong machine. Over two months I hit this on our three-node Docker Swarm cluster more times than I want to admit, and it taught me to stop trusting any deploy signal that doesn't come from inside the running container.

The setup

We run a handful of production and UAT apps (Hoadhaa, a sports-federation portal, a BoltHR UAT copy, an internal telemetry portal) on three VPS nodes. They form a single Docker Swarm, share a replicated GlusterFS volume, and run Dokploy on top as the deploy panel. Dokploy is a great self-hosted PaaS: point it at a Git branch, it builds a Docker image, it updates the Swarm service. Cheap, simple, and it mostly feels like Heroku.

The catch is in one field: every app had its registry set to none. With no registry, Dokploy builds the image locally on the node running the panel and tags it app:latest. On a single server that's fine. On a cluster it's a time bomb.

How a green deploy runs old code

Here's the sequence, and it plays out exactly the same way every time:

1. Dokploy builds app:latest on node1, which runs the panel.

2. It tells Swarm to update the service to app:latest.

3. Swarm schedules the new task wherever it likes, say node3.

4. node3 already has an image called app:latest: the one from the previous deploy.

5. The container starts happily on the stale image. Health checks pass, because the old code is perfectly healthy.

6. Dokploy reports success.

No error at any point. A bare :latest tag with no digest pinned in the service spec means "whatever this node calls latest", and each node has its own answer.

The worst version I hit was a Laravel backend where only the web service runs migrations on boot. Both web replicas landed on nodes with stale images. The deploy was green, the app served traffic, and the schema change never happened. The only honest signals were migrate:status and the migrations table. If the next release had depended on that column, we'd have had a production incident that looked like it came out of nowhere.

Why unpinning made it worse

At first this bit only now and then, and the reason was a different mistake. Most services were pinned to node1 (node.hostname==node1), the same node that builds. Build node and run node were the same machine, so the missing registry couldn't hurt us. It also meant node1 was a single point of failure for every API we hosted.

When I removed the pins to get real high availability, the trap went from occasional to every single deploy of an unpinned app. That was the lesson that stuck with me:

> Unpinning a service trades a single point of failure for exposure to the missing registry. The two problems are linked, and the honest fix for both is a registry, not re-pinning.

Why we didn't just add a registry

We tried. A GHCR registry entry already existed in Dokploy, just not attached to any app. Attaching it failed: the stored token was a fine-grained GitHub PAT, and those can read GHCR packages but not push to them. Publishing needs a classic token with write:packages. The registry entry also had an empty image prefix, which would have produced invalid ghcr.io/ references.

The order I wrote down for doing it properly is worth sharing, because getting it wrong breaks every deploy at once:

1. Fix the token and set the image prefix.

2. Attach the registry to the worker first (no traffic), deploy, and confirm other nodes actually pull the new reference.

3. Then the scheduler, then web.

If you attach all three before you've proved a push works, every deploy fails after a successful build. Until that happened, I needed a routine that worked without a registry.

The interim fix: GlusterFS as a poor man's registry

Every node already mounts a shared GlusterFS volume, so I used it as the transport. A small image-sync script runs every 15 minutes on all nodes. The builder docker saves locally built images to the shared volume, and the other nodes docker load any they're missing. Public images (MySQL, Postgres, Metabase) are skipped, because each node pulls those itself.

A cron job on its own isn't enough, because it can run before the build finishes. So a deploy of any unpinned app became a three-step operation:

# 1. deploy via panel or webhook; wait for "done"
2. replicate the image to every node
sudo image-sync.sh            # on the builder: export
sudo image-sync.sh            # on each other node: import
3. move the task onto the new image
sudo docker service update --force <service>

With update-order: start-first, the forced update has been zero-downtime every time: in one run 45 out of 45 probes returned 200 through the rolling restart.

The manual version, for when the script isn't there, is just as short:

docker save -o /mnt/shared/transfer.tar app:latest   # on the builder
docker load -i /mnt/shared/transfer.tar              # on each other node
docker service update --force <service>
rm /mnt/shared/transfer.tar                          # it's hundreds of MB

Verify inside the container, not by image age

This is the part I'd tattoo on a junior engineer. Image age on the builder proves nothing. Image IDs across nodes prove nothing either. Version skew between Docker engines made save/load re-derive image IDs, so the same build showed different IDs on different nodes.

What works is checking the running container for something the commit actually changed:

docker exec <container> sh -lc 'test -f app/Enums/Payments/BankEnum.php && echo NEW || echo OLD'

One gotcha caught me once. If you grep for the absence of old code, your own commit comments can betray you: a comment quoting the old expression matches, and the check reads "still stale". Grep for live constructs (a new constant, a new class) instead.

Making the app tell on itself

Running docker exec by hand after every deploy doesn't scale, so in Hoadhaa I made the application report its own build. The Dockerfile takes ARG BUILD_SHA and bakes it into APP_BUILD_SHA. The web, worker and scheduler containers write heartbeats to a system_heartbeats table every minute with their role, build SHA, memory and disk. An admin System health page compares them.

Choosing the "expected" build was trickier than I expected:

• •The expected build is the SHA of the process answering the request, not the majority. If two of three containers are stale, a majority vote calls the stale build correct.
• •A container is marked replaced (not flagged) when a same-role container on the expected build started at least 120 seconds after it. That hides normal rolling-restart noise.
• •Replicas from the same roll that started seconds apart on different images stay flagged. That's exactly the signature of the registry trap.

The same SHA goes to Sentry as the release, so errors are tagged with the build that actually threw them rather than the build we think we shipped.

The hidden cost: disks fill up

Every sync leaves the previous build dangling on every node, and nothing cleans it up. One morning a node's root filesystem was at 96%; one more deploy would have filled it. Now a dangling-image prune is part of the routine (it's safe: tagged images and running containers are untouched), followed by a check that the freshly synced image survived:

docker image prune -f
docker image inspect app:latest --format '{{.Created}}'

One prune reclaimed over 30 GB on a single node. Shrinking the images helped too: dropping the .next/cache build cache and dev dependencies took one Next.js image from 2.9 GB to 1.5 GB, which halves the cost of every sync.

Takeaways

• •A bare :latest with no registry on a multi-node Swarm is a correctness bug, not a performance one. Each node has its own idea of "latest".
• •Pinning hides the bug; HA exposes it. If you unpin for availability, fix image distribution first.
• •"Deploy succeeded" means "the orchestrator accepted the spec". Verify the code in the running container.
• •Make the build identity observable (SHA in an env var, in heartbeats, in your error tracker), and compare against the process answering the request, not a majority vote.
• •Prune after every sync. Image distribution without garbage collection is just a slow disk-full alert.