Pour one out for the homies.

The prologue

A few days earlier, I pulled a small HP desktop out of storage. It had a six-core processor, room for more memory, and an obvious destiny: a dedicated server for a survival game a few friends and I wanted to play.

The house had no power supply for it. Money was tight, and a game server could not justify sixteen dollars until payday, eight days out. The HP went onto the project bench as an aspirational reminder.

4:17 a.m.

The family's main server was a fanless mini PC, and it ran most of what you see on this site: these pages, Fed Forager, the media library, the download tools, the family DNS and the notification relay. At 4:17 a.m. it dropped off the network without saying goodbye. The other cluster members noticed it had left, but it had not shut down; it had simply stopped.

It did not come back. Pressing the power button did nothing at all: no light, no fan. Every other machine in the rack had stayed up all night, so it was not the house's power. The board itself had given up.

Home Assistant, which runs the house, lives on a different machine and kept running. Only its remote entrance broke, because that front door lived on the box that died.

What survived

The good news arrived quickly. The media and family files live on two external drives, not inside the dead box. The nightly backups had finished at 4:11 a.m., six minutes before the failure. Almost nothing written that night was lost; the media server lost about a day of watch history.

By mid-morning the drives were plugged into another node, a slimmed-down copy of the main services was running there, and this site showed an outage notice that began, fittingly, "Pour one out for the homies."

The middle part

You can probably picture the rest of the day:

• Ordering a same-day power supply for the HP, then refreshing the delivery window like it owed me money.
• Discovering the HP had only half the memory I remembered, then salvaging two memory sticks from retired laptops.
• Finding that one of those laptops had a swollen battery, the kind people politely call a "spicy pillow". It spent the afternoon in a stockpot in the bathtub before being dropped, with some ceremony, into the recycling bin at the hardware store.
• Moving the dead machine's own system drive into the HP, so the replacement would boot as the old server, with everything exactly as it was.

The power supply arrived at night. The HP's video output turned out to be DisplayPort, so I went looking for a DisplayPort cable in the box of uncommon cables.

In that same box, coiled neatly beside the cable, was the HP's original power supply.

The ghost in the rack

The HP booted the old server's drive on the bench. One renamed network port later, it was back on the network. I shut it down, carried it to the rack, and powered it on.

It answered for six seconds, then vanished.

It came back. Six seconds, gone again. A new cable, a different patch-panel port, a different switch port, a different cable run: same result. A small script beeped low while the server was missing and high while it answered, so I could hear the problem from the rack. Mostly it beeped low.

The logs eventually told a strange story: the machine was rebooting itself about every 84 seconds, with no crash report and no error, as if someone kept pressing a reset button. On the bench it was fine. On the rack it was not. The difference was not the network, the power or the location. On the bench, a monitor was plugged in.

Unplugging the monitor on the bench, with a live log streaming, confirmed it: a few seconds later the server simply ceased to exist, mid-second, without a word. With no display attached, this particular little HP resets itself.

The fix costs about five dollars: an HDMI dummy plug, a tiny adapter that tells the computer a monitor is attached. I happened to have one within arm's reach. Seven minutes later it had not missed a single ping.

10:57 p.m.

The drives moved over, the storage pool came up, and the original server started exactly where it had stopped that morning: 51 services, the real homepage instead of the outage notice, backups armed for the night. The external drives now run at USB 3 speed instead of USB 2, so the old server came back faster than it left.

What I'm keeping

• Backups should not live only on the machine they protect. Both nightly jobs ran on the box that died. They worked, but only because they finished before it did.
• Back up the small things. The big data was covered. A handful of tiny configuration files were not, and those turn a restore into a rebuild.
• Soak-test headless. If a machine will live without a monitor, test it without a monitor for a few minutes before it goes in the rack.
• Check the cable box first.

The spare HP did eventually become a server. It just wasn't the one it was meant to be. The game server is still on the list.

Related: Backups & recovery · When free space lies