A backup you never restore is a big folder with a reassuring name

Until late August the homelab had no backups of any kind. A failure of the system disk would have taken the firewall configuration, the password vault and every container. So I set up Proxmox Backup Server, a daily job at 02:30 across all nine machines, with a retention of seven daily, four weekly and six monthly. The dashboard said yes to everything.

Before signing the system off I did what almost nobody does: an actual end-to-end restore, checking that what came back was usable.

The restore

I restored the password manager’s container to a new container, without touching the production one and without starting it automatically: it would inherit the original’s network configuration and there would be two machines asking for the same IP. Then I started it by hand and looked at the data that mattered:

db.sqlite3          286720 bytes   <- the vault
db.sqlite3-wal           0 bytes   <- no half-written transactions
rsa_key.pem           1675 bytes   <- signs the session tokens

Each file confirms a different thing. The WAL at zero bytes was precisely my doubt about snapshot mode, which takes the copy with the container running: it came out clean. And without the RSA key, every device would have to authenticate again. I destroyed the test container and felt reassured. Rightly, but less rightly than I thought.

The gap

The next day I uploaded the first 4,133 photos to Immich. And before deleting them from the iPhone, I checked that they were backed up.

They weren’t. The signal was the duration: a manual backup of the Immich container finished with TASK OK in three minutes. For 44 GB over a USB disk that was impossible. The datastore took up the same space as before the library was uploaded. And the log said so, in a line that’s easy to skip over among the others:

INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/library') from backup (not a volume)

The hypervisor’s backup tool works on volumes managed by Proxmox. The photo library wasn’t in one: it was a bind mount, linked from the big drive into the container precisely so it wouldn’t take up space on the system disk. And those mounts don’t go into the backup. Ever. It isn’t an option: it’s by design. I tried adding backup=1 to the mount point and it did nothing, because that option only applies to volumes.

No error, no warning, nothing in red. The job finished successfully because it was doing exactly what it knows how to do. The only thing missing was me assuming it covered something it had never promised to cover.

The fix

Three options: the PBS client from the host backing up the directory straight into the datastore, rsync to another disk, or converting the mount into a managed volume. I chose the first because it reuses the infrastructure that was already there: proxmox-backup-client scheduled with a systemd timer at 03:15, an hour after the general job, with its own retention.

The same reasoning applied to the films folder, which is also a bind mount. There I left it unbacked on purpose: if they’re lost, they get downloaded again. If the photos are lost, they don’t.

The general shape of the mistake

This failure isn’t about one particular tool. It’s about the fact that green does not mean complete. An automated job tells you it did its task, not that its task is the one you think it is.

Four things I’m taking away:

1. A backup that finishes fine doesn’t mean it contains what you think. You have to check what it backs up, not only that it runs.
2. Duration and size are diagnostic signals. If a 44 GB backup takes three minutes, something is missing. Numbers that look too good deserve a check.
3. du -sh on the datastore is the most direct check. If it doesn’t grow when it should, the data isn’t going in.
4. Verify before destroying the original. I checked before deleting the photos from the phone. Had I deleted them first, the only copy would have been the USB disk, unbacked and unknowingly so.

The one check that can’t be faked is the restore. Everything else is trust. And now with a caveat: having a backup that runs is not the same as having the data that matters backed up.

What’s still wrong

Every backup lives on the same physical disk, in the same house, as the original data. Against a system-disk failure, an accidental deletion or corruption, I’m covered. Against a fire, a burglary, a power surge or a failure of the backup disk itself, I’m not.

I’m writing it down because the fix is a sixty-euro drive plugged in once a month, and putting it in writing makes it more likely I actually buy it. And because I’ve scheduled a test restore every three months: it’s the one most easily forgotten and the only one that proves the system still works.