I moved my home media server off a single Docker Compose file on my laptop and onto a Proxmox box, one service per VM. That felt like a clean step up. Then I spent a night staring at a media server that listed every film perfectly and refused to play any of them with a flat "source error." The fix wasn't in the media server. It was three mounts down, in a chain I'd built myself without realising how many places it could quietly break.

This post covers the storage design, the three-layer mount chain that came out of it, the self-healing timer that finally made it boring, and what I'd change if I were starting over.

The constraint that dictates everything: one path

A self-hosted media setup usually has a few services that touch files: something writes new files into an incoming folder, something organises them into the library, and the media server reads the library. The trick that keeps this fast is hardlinks. When a file is "moved" into the library, it isn't copied. A second directory entry points at the same inode, so the move is instant and uses no extra disk.

Hardlinks only work inside one filesystem, and the services only agree on where things are if they all see the same path. In the original Compose stack that came for free, because every container bind-mounted the same host folder at /data. Split those containers into separate VMs and you lose it by default. Each VM has its own disk, and a hardlink can't cross VMs.

So the whole design came down to this: one storage VM exports a single NFS share, and every VM that touches files mounts it at the identical path /data.

storage VM:  /data  (NFS export)
   -> /data on the media server, the ingest service, the library managers
/data/incoming   new files land here
/data/movies     library
/data/tv         library

The services that never touch files, like the request UI and the reverse proxy, deliberately don't mount it. A hard NFS mount couples a machine to storage availability, and there's no reason to give a stateless web UI that failure mode.

Before building anything else I checked the one assumption that would have sunk the design: do hardlinks survive NFS between VMs?

ln /data/incoming/test /data/movies/test
stat -c '%i %h' /data/incoming/test /data/movies/test
same inode, link count 2

Same inode, link count 2, atomic rename worked, and a write on one VM was readable on another. If that had failed I'd have gone back to a single big VM.

Two smaller details mattered just as much:

• •Consistent identity. Every VM got a media user and group with the same fixed UID and GID, and every container runs with that PUID/PGID. My laptop stack had used the macOS defaults (501:20), which don't exist on Linux. Mismatched UIDs give you files nobody can rename.
• •async on the export. On a single 7200 rpm disk it's a big throughput win. The trade-off is that writes can be lost on an unclean power-off. For a library that can be rebuilt, that's fine. If it ever holds anything irreplaceable, I'll revisit it.

Why the media server ended up in an LXC

The media server is the one service that benefits from hardware transcoding, and the box has an Intel iGPU with Quick Sync. Passing a GPU render node into a full VM is a project. Passing /dev/dri/renderD128 into an LXC container is a couple of config lines. So the media server became a privileged LXC with nesting enabled, running Docker inside.

That choice is what created the chain:

1. The Proxmox host mounts the NFS share at /mnt/media (a plain fstab entry).

2. The LXC config bind-mounts host /mnt/media into the container at /data.

3. Docker inside the LXC bind-mounts the container's /data into the media server container.

Each layer can fail on its own, and they fail in different ways.

The failure: "library lists, playback says source error"

This is the signature, and it's a good one to learn. The library listing comes from the media server's database. Playback opens the actual file. If /data is empty, the library still looks perfect and every play fails.

What had happened at boot:

• •The storage VM is a guest of the same host that mounts it. At boot the host tried to mount the share before the storage VM was serving it. The mount timed out, and systemd didn't retry.
• •With nothing mounted, /mnt/media on the host was just an empty directory on the root disk. The LXC bind-mounted that empty directory. Docker bind-mounted the empty directory. Everything was "up."

Fixing layer 1 by hand made the files show up in the LXC straight away, because bind mounts propagate. The Docker container still saw nothing. It holds whatever it bound when it started, so layer 3 needs a docker restart. The same stale-bind trap showed up on every other VM that runs Docker over NFS. A service there was reporting "missing files" for things that were plainly on disk, until its container was restarted.

The rule I took away: after restoring an NFS mount, restart every container that binds it.

Debugging NFS without hanging your shell

A hung NFS mount puts ls, df and even mount into uninterruptible sleep (D state). Not even timeout can kill a process in D state. I lost two long tool calls that way before I changed habits:

• •Check mounts with grep /data /proc/mounts, never mount | grep.
• •Never ls a mountpoint you suspect is stale.
• •Wrap remote checks in timeout anyway. It won't save you from D state, but it saves you from everything else.
• •Check every NFS client, not just the one that complained. The same boot left one other VM unmounted while two were fine.

The fix: a boring self-healing timer, not clever ordering

The obvious fixes all had sharp edges:

• •x-systemd.automount turns the mount into an autofs trigger, and binding an autofs trigger into an LXC is flaky.
• •Ordering the container after the mount risks deadlocking against pve-guests.service, which starts both the storage VM and the LXC. The NFS server is a guest of its own client, so the boot order is circular.
• •Editing fstab cleverly on a box where a typo means no network was not something I wanted to do at 2am.

So I did the dull thing. A small script and a systemd timer on the host, plus the same pattern on each NFS client VM:

1. Wait until the storage VM answers on port 2049.

2. Mount if it isn't mounted, and force-remount if the handle is stale.

3. For each : pair in a list, check whether the bind is empty and restart that Docker container if it is.

[Timer]
OnBootSec=45s
OnUnitActiveSec=5min

It runs 45 seconds after boot and every five minutes after that, so it covers a cold boot and a storage VM reboot in the middle of the day. The fstab entries stay plain defaults. The host may still log one failed mount at boot, and the timer heals it shortly after. On the next cold boot the mount failed exactly as before, and the timer mounted it and restarted the right containers with nobody touching anything. That's the result I wanted.

The one maintenance cost: whenever I add another Docker-over-NFS service, it has to be added to that pair list.

The honest weak spot: one disk

All of this runs on a single 1 TB 7200 rpm disk: the hypervisor, every VM disk and the media library. That has costs you can see:

• •Starting many VMs at once stalls cloud-init on several of them. A qm reset clears it, but the real fix is staggered starts.
• •Downloads, transcoding and nine guests all compete for one spindle.
• •A single disk failure takes out the host, every VM and the library together.

If I were starting again, in order:

1. An SSD for VM disks. That's the biggest real-world improvement for the money.

2. A second disk and a ZFS mirror for the library.

3. Backups sent off the box. A backup on the same disk that's failing isn't a backup.

Takeaways

• •Pick the shared path before you pick the topology. For hardlink-based media workflows, an identical /data everywhere is the design. Splitting into VMs means you have to rebuild it deliberately, usually with NFS.
• •Test the assumption the design rests on first. Ten seconds of ln and stat told me the architecture would work.
• •Know your mount chain and how each layer fails. "Lists but won't play" means an empty mount, and Docker binds don't heal on their own.
• •Prefer a dumb, idempotent healer over clever boot ordering when the dependency graph is circular.
• •Spend your first upgrade money on the disk layout, not on more services.