Three times it "looked configured": verify by reading the result, not by assuming

The homelab has eighteen documented mistakes. Most of them are different from each other. Three aren’t: they’re the same mistake with three faces, and by the third it stopped being a coincidence.

The pattern goes like this: you run a command to configure something, the command doesn’t error, you consider the change done, and the system sits there looking configured without being so. Nothing breaks at the time. It breaks days later, or it never breaks because the protection you thought you had never gets tested.

Case 1: ip_forward went back to zero on its own

I enabled packet forwarding on the host so Tailscale could route into the VLANs. I did it with sysctl -p on the fly and it worked. Days later, after several reboots during the network redesign, remote access to the lab VLAN started timing out. The route was advertised and approved, the host could reach the machine, the firewall rules were correct.

sysctl net.ipv4.ip_forward
# 0  <- this is the problem

The line had never been written to /etc/sysctl.conf. sysctl -p applied whatever was there at the time; a reboot restored the kernel’s default, silently, and it went unnoticed until that specific route was needed.

Case 2: the udev rule that would never have fired

The external disk kept falling asleep. hdparm -B 255 -S 0 fixes it, but it’s lost on reconnect, so you need a udev rule that reapplies it every time the disk shows up. The first attempt filtered on the manufacturer:

ENV{ID_SERIAL}=="*Verbatim*"

Seems reasonable. But ID_SERIAL reports the actual internal drive, a Toshiba, not the enclosure; the string “Verbatim” only appears in ID_USB_SERIAL:

ID_SERIAL=TOSHIBA_DT02ABA600_…
ID_USB_SERIAL=Verbatim_Desktop_HDD_3.0_…

That rule would have sat in the system forever, looking configured, without running once. The disk would have kept falling asleep and I’d have looked for the cause anywhere else.

Case 3: the sed that edited a comment

To make the LVM thin pool auto-extend you change a threshold in /etc/lvm/lvm.conf. I ran a sed looking for thin_pool_autoextend_threshold = 100 and replacing it with 80. It ran without error and changed nothing useful: in that file all those lines are commented out, because they’re the built-in documentation, not active configuration. The sed edited an example line inside a comment.

Only by checking the file with grep did it become clear that every match started with #. The fix was to find the real activation section and insert the lines inside it, then check with lvs that the file was still well-formed.

The pattern

All three commands “worked”. None of them lied: they did exactly what they were asked. What failed was the assumption that what I asked for was what I wanted.

The habit that catches all of them

After configuring something, verify by reading the result, not by assuming the command did what it looked like. sysctl net.ipv4.ip_forward after sysctl -p. grep on the file after sed. Read the device’s real properties before writing the rule.

And when it’s an automation, the definitive test is a different one: force the opposite state and check that it reverts. With the disk:

hdparm -B 128 /dev/sda           # force the opposite state by hand
udevadm trigger --action=change /dev/sda
sleep 2
hdparm -B /dev/sda               # if the rule works: off

It returned off. That proves the rule actually runs, not that the value simply stayed as it was. It’s the difference between observing a coincidence and demonstrating a cause, and it’s a test that takes ten seconds.

I’m writing it down because the pattern isn’t about Linux or homelabs. It’s about any system where a change is applied in one place and read in another: environment variables, configuration files with commented-out defaults, migrations that “ran”, permissions that “were granted”. If you haven’t read it back afterwards, you don’t know.