The Log Is the System

There is one machine in my house that everyone depends on and only one person can fix. It serves the films, the photos, the scanned paperwork, the family's file sync and a wiki. When it goes down I do not get a ticket, I get asked why the TV isn't working. After three years of running it I have concluded that the hard part of a home server is not building it. It is the paperwork.

Not documentation in the corporate sense, but two specific artifacts: a runbook written for somebody who isn't me, and a dated log of everything that has broken. Both have paid for themselves several times over, and the log has done something I didn't anticipate. It has repeatedly proved me wrong.

What It Actually Runs

One box, Ubuntu LTS, about twenty-one containers. Plex and Jellyfin, the usual media automation, Paperless for scanned documents, Nextcloud with MariaDB behind it, a download client, a wiki, and a Postgres instance for the document archive.

Every one of those is a container, on purpose, so that each service gets its own bucket sharing no files with any other and a mess in one can't contaminate its neighbours. In front of them sits Traefik as a reverse proxy, which matters because several domains point at this one machine and something has to sort out which request belongs to which service. Domains are registered at Hover and their DNS lives at deSEC; Traefik talks to deSEC directly to provision certificates, so adding a service is a config change and nothing more.

Storage is ZFS, which I have written about separately. This post is about everything around it.

A Runbook for Someone Who Is Not Me

The most useful page I have written about this machine contains almost no technical content. It is addressed to whoever has to get into the server when I'm not available (travelling, in hospital, or worse) and it opens with plugging in a monitor and keyboard, and where to find the password.

It also contains this line, which I think about a lot:

sudo asks for the password and shows no output as you type — that's expected, keep typing.

That sentence is in there because a silent password prompt is alarming if nobody has told you it's normal, and because someone stuck at that prompt at a bad moment will conclude the keyboard is broken and give up. You stop seeing it once you know it, which is why it needs writing down.

The rest of the page keeps to the same register: what ZFS is in two sentences, three commands that show what is plugged in and where it is mounted, and the fact that each drive's serial is printed on its own label so the right disk gets pulled. No architecture, no rationale, just enough to act.

Writing it changed how I build things, too. Anything I couldn't explain in a paragraph to a non-sysadmin was something I'd made needlessly clever.

The Day Everything Happened at Once

In June I upgraded from one Ubuntu LTS to the next, on a host that had been quietly wedged for weeks. Four things went wrong in sequence, and three of them were interesting.

The first reboot hung at firmware level on a kernel that had never been booted. Userspace shut down cleanly and the machine just never came back. A hard power-cycle fixed it, and the cheap insurance for next time is reboot=force,warm on the kernel command line (worth having on any host you can't easily put a monitor on).

The SSH server's post-install failed because its socket unit was masked. The new release moves SSH to socket activation, and anything that had ever masked that unit blocks the package from configuring. This is the one that could have locked me out of my own machine, and the discipline that saves you is boring: always verify the new SSH daemon works in a second session before closing the first.

The conffile prompts caught a real regression. The upgrade offered to replace nine config files, and working through the diffs by hand recovered 56 hardening sysctls that had been silently lost (kernel lockdown settings, swap tuning, the entire network-stack hardening set). Had I hit "accept the maintainer's version" nine times, the machine would have come up meaningfully less hardened than it went down, and nothing would have told me. Those .dpkg-old files are your configuration. Read them.

The comic postscript: restoring those sysctls also restored net.ipv4.ip_forward=0, which broke Docker's bridge networking. Every container was running and none was reachable. So the fix for one silent problem created a loud one. The better failure mode, but it still cost me an evening.

When Monitoring Cries Wolf

For eight months my server mailed me every day about a failing drive. It wasn't failing.

The attribute in question counts sectors a background scan could not read. It read 32 in January and 32 in September, unchanged across roughly 5,700 power-on hours, with zero pending sectors, zero reallocations, a passing extended self-test taken after those sectors appeared, and scrubs repairing nothing. The drive firmware never decrements that counter, so the number is a latched high-water mark.

The cause was one character. The smartd directive -U 198 reports the attribute whenever it is non-zero, forever, with no notion of a baseline; -U 198+ reports only on an increase. One suffix separates "tell me if this gets worse" from "tell me this exists, daily, until the heat death of the universe."

Same week, same file: both NVMe drives had been mailing about reaching a "critical" temperature of 55°C. That threshold was mine, not theirs, applied blanket to every device on the host whether spinning or solid-state. The drives' own critical threshold is 85°C, and their counters for time spent in warning or critical state both read zero. They have never once been outside spec. They idle around 40°C and burst to 70 under write load, the top of their normal operating range and fifteen degrees below where the drive itself would complain.

Neither alert reflected a degrading drive. Both had trained me to ignore mail from the one channel that would tell me if something was actually wrong. I now think an alert that has fired daily for eight months without a corresponding action is worse than no alert at all, because it consumes the attention a real one needs.

The Near-Miss That Bothered Me Most

Investigating those alerts turned up something worse than either of them. The config file I was editing had been hand-modified at some point and no longer matched the Ansible role that generates it, and that role templates the file unconditionally.

The next routine playbook run would have silently overwritten the lot: every per-device entry and all the temperature tracking, gone, with no error and nothing in the output to notice.

This is the real cost of configuration management and it's rarely stated plainly. Once a file is managed, editing it on the host isn't a shortcut; it is a change with a delayed-action deletion attached. The machine will keep your edit for days or months and then discard it at a moment unrelated to anything you were doing, which makes the eventual breakage almost impossible to connect back to its cause.

I caught this one by accident, while looking into something else. That is not a system, that is luck. The fix was to put the per-device entries into the role properly, but the lesson I actually took away is that "I will just tweak it on the box for now" is a promise to future-me that I have no mechanism to keep.

When the Failure Hides

My favourite failure of the year presented as "Plex cannot find any files."

Nothing was wrong with Plex, and nothing was wrong with the data. An encrypted dataset simply had not mounted at boot: its key-load unit ran before the pool finished importing, and failed. Its mountpoint showed an empty directory, and every container bind-mounting a path underneath it came up pointed at that empty stub.

The part that took longest to understand is that mounting it by hand didn't work either. The mount unit is bound to the key-load service, so with that service in a failed state systemd tore the mount straight back down a few seconds after each attempt, with a log line explaining exactly what it was doing (which I read four or five times before it registered).

Recovery is to start the key-load service, then the mount, then restart the affected containers. That last step isn't optional, because a bind mount established against an empty directory doesn't pick up a later mount on the host. The container keeps looking at the stub until it is restarted.

Two things made this worth the trouble. First, it is a known upstream defect (openzfs/zfs#11248, labelled against encryption and systemd, and sitting stale). The coupling that causes it is intentional, since a failed key-load shouldn't proceed to a doomed mount, and it is still present across current releases, so there's no version to upgrade into. A workaround is the expected remedy here.

Second, the durable fix taught me something about writing repair code. The unit that now handles this waits for the import to finish, clears the failed state, loads the keys, mounts, and then restarts every running container whose bind mounts overlap the freshly-mounted path, discovering that overlap by inspecting the containers at run time. There's no hand-kept inventory to forget to update. When I tested it, it correctly found three affected containers, and I had only remembered one of them.

Any repair script that depends on a hardcoded inventory will eventually be wrong, because the inventory changes and the script doesn't.

Correcting the Record

The incident log now contains several entries that begin, in effect, "the note above is hereby corrected." A drive I called dead was healthy. A counter I described as climbing hadn't moved in eight months. A kernel I blamed for tipping over marginal hardware had done nothing of the sort.

Those corrections are, I think, the most valuable lines in the whole document. A log that only records what happened is a diary. A log that records what I believed at the time, and then what was actually true, is a record of how I get things wrong, and the failure modes repeat. I reach for the mechanical explanation over the electrical one. I trust a status field as a diagnosis rather than a symptom. I build mitigations on a theory before I have tested the theory.

Knowing that about myself has been worth more than any individual fix in there.

The machine is healthy as I write this, with pools online, scrubs clean, containers up, and a list of open action items ranked by priority that I work through when I have an evening. That list is never empty and I've stopped expecting it to be. A home server isn't a project you finish. It is a small ongoing obligation you take on in exchange for owning your own data, and the log is what turns six separate incidents into something that resembles experience.