The Night My Homelab Went Dark: A Wedged NIC, a Smart Plug and an Impostor Laptop

At 1:19 one morning my homelab disappeared from the network. The Proxmox host and all nine guests went dark at once. No ping, no SSH, no web UI. When I got to the console, the machine was fine: clean journal, normal load, nothing crashed. It just wasn't on the network any more.

Working out why took me through three unrelated problems: a network card that wedges itself, a router handing out my servers' addresses to phones, and a smart plug that sometimes answered for my storage server. There was also a fourth surprise I'll come to. Each one was hiding behind the others.

Symptom one: the host is alive but nobody can reach it

The kernel log had the answer, repeated every two seconds:

e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang
  TDH <46>  TDT <1>  next_to_clean <45>

119 of those in four minutes, until I rebooted by hand. What mattered most was the line that never appeared: no "Reset adapter." The e1000e driver noticed the transmit ring was stuck and never recovered it.

A few things this was not, which saved me time once I saw them:

• •Not the cable or the switch. Link detected: yes the whole time, no link flaps. The physical link was fine. The NIC's transmit queue was stuck.
• •Not the disk. smartd announced at boot that the ATA error count had gone from 0 to 8, which looks alarming. All eight errors were logged at power-on hour 1,515, and the drive is past 17,000 hours. They were about two years old. smartd had lost its saved state and was reporting history as news. Reallocated, pending and CRC counters were all zero.
• •Not a guest problem. Every VM and container bridges through that one physical port, so one wedged NIC is a total outage. That's obvious once you see it, and easy to forget when you're staring at nine dead services.

What triggered it

When it hung I was running two SSH fan-outs at once, each talking to nine hosts in parallel. That's a short, bursty spike of small outbound packets. This family of Intel onboard NICs has a long history of hangs under that kind of load, usually linked to hardware offloads such as TCP segmentation offload.

The fix, and the fix I didn't make

I turned off the offloads on that interface:

ethtool -K eno1 tso off gso off gro off

I put it in a small oneshot systemd unit ordered Before=pve-guests.service, so guests always start on a NIC that already has the change. A periodic health script reapplies it too.

Two things I deliberately didn't do:

• •I didn't edit /etc/network/interfaces. A mistake there causes exactly the outage I was trying to fix, on a box I'd need to walk over to.
• •I didn't run ethtool -K in my SSH session. Changing offloads bounces the link, which kills the session halfway through. Starting it as a service with systemctl start --no-block let the change finish while my connection dropped and came back about fifteen seconds later.

I also changed how I work. On this box, remote sweeps now run one host at a time or staggered, never as a parallel blast. That costs a few seconds and avoids a 2am trip to the server.

Symptom two: the storage share that sometimes wasn't there

With the NIC sorted, an older problem came back into view. Every so often after a reboot, the NFS share the media VMs depend on failed to mount. It worked sometimes, which is the worst way for something to fail.

I cleared the ARP entry for the storage server's address and pinged it, one address at a time and on its own. Twelve times. Nine replies came from the storage VM's MAC and three came from another device.

The OUI on that MAC belonged to an IoT vendor. It was a smart plug or bulb somewhere in the house, with no open ports, talking only to its cloud. It roamed between addresses and sometimes landed on the one my storage server uses as its static IP. When it held that address at the moment the host tried to mount NFS, the mount went to a light bulb and timed out.

One warning from this part of the hunt: don't trust ARP readings taken during a parallel ping sweep. My first sweep of the whole subnet showed the wrong MAC for the storage server and briefly pointed the blame at the storage VM itself. Clear the entry, ping one IP, read it, and repeat.

Root cause: the DHCP pool was the whole subnet

The ISP-supplied router's DHCP pool covered every usable address in the subnet. My servers set their addresses statically on the machines themselves, so the router never knew those addresses were taken. As far as it was concerned, even the hypervisor's own address was free to lease to the next phone that joined the Wi-Fi. With a one-hour lease, that churn never stopped.

The fix was one screen in the router's admin UI. I moved the pool start and end so that a block at the bottom of the range is never handed out, and all servers live in that block. I reloaded the page afterwards to confirm the setting had stuck.

Until I did that, I relied on a stopgap I'd recommend to anyone in the same spot: a permanent ARP entry on the hypervisor for the storage server. It's only safe if it's done carefully:

• •The MAC is read live from the VM's config every time, never hardcoded. Rebuild the VM with a new MAC and a hardcoded pin silently breaks NFS forever.
• •A runtime ip neigh entry doesn't survive a reboot, and that's exactly when you need it. So the same health script that reapplies the NIC settings also re-pins the entry every five minutes. I tested it by deleting the pin and watching the timer put it back.

The fourth surprise: my laptop was impersonating the server

While I was looking at address conflicts, I found my laptop sitting on the media server's static IP. Its private Wi-Fi MAC meant it didn't even look like a Mac in the ARP table.

It was also still running the old local Docker version of the media stack, which included a different media server listening on the same port. The reverse proxy forwarded the public hostname to that IP and port. So whenever my laptop was awake, it was serving my public media site instead of the server.

That explained two complaints I'd been treating as separate bugs: the wrong media server sometimes loading, and playback errors that came and went. Laptop asleep, requests hit the real server, which at the time had its own storage problem. Laptop awake, everything streamed fine from the wrong machine. It was a very convincing fake of an intermittent NFS bug.

The fast way to tell them apart was the version string on the server's public info endpoint, because the two products report very different version numbers. I stopped the old stack (it was set to unless-stopped, so an explicit stop survives reboots) and moved the laptop off the server range.

What I'd tell myself before building it

• •Give servers a range the DHCP server will never hand out, using the pool range, reservations or both. A static IP set only on the host is a promise the router doesn't know about.
• •A single onboard NIC carrying every guest is a single point of failure worth knowing about. If I add hardware, a second NIC (ideally not an onboard e1000e) is on the list.
• •Be gentle with bursty parallel SSH against hosts behind consumer NICs.
• •Write down the cold-boot timeline. This box takes about six minutes from power-on to the public site answering, and guests start in a staggered order. "Everything is stopped" two minutes in is normal. Knowing that stops you from intervening in a boot that's working.

Takeaways

Intermittent problems are usually several problems layered on top of each other. Here a wedged NIC, an over-generous DHCP pool, a roaming IoT device and a forgotten laptop container each produced symptoms that looked like the others. What got me through was narrowing each measurement until it could only mean one thing: one IP at a time, version strings instead of "it loads," driver logs instead of guesses. And for the fixes, prefer small, idempotent, self-reapplying changes over clever edits to the files that keep the machine on the network.