This article is about two desirable traits in system upgrades: atomicity and being able to roll back.
- Atomicity: the upgrade is either applied in full or not applied at all. The system is not left in a broken state because, e.g., “the package manager updated half of the files and then died”. This includes situations like a power failure mid-upgrade.
- Rollback-capable: if a system upgrade fails, or doesn’t work, it’s still possible to use the device in a previous system state.
These two traits dramatically increase the reliability of a device that’s updated in the field, where the risk of an unusable device after an upgrade should be as minimal as possible.
A common anti-pattern: immutable images
A common trend nowadays is to entangle these two features with immutable system images: the whole OS ships as an image, the entire image is downloaded (or perhaps a delta, layers, etc.) rather than overwriting individual files, ensuring atomicity. The old image is retained, guaranteeing the ability to roll back the system.
I have a strong distaste for this “immutable system image” approach: the design aligns well with a movement to prevent the user from understanding and experimenting with the system, reserving that prerogative for the system vendor alone. It aligns with the movement against general purpose computing, and with the idea that users shouldn’t be able to tinker with their own devices, or modify them in any way; that the ultimate control of the device should lie with the vendor alone.
Ironically, the idea of immutable system images has permeated even into circles which stand in favour of hackable user-controlled devices: propaganda has convinced folks that this is the only way to obtain transactional updates, and individuals who actually want user-controlled devices regrettably still push for immutable image-based systems.
Even while some of these systems are open source and can be studied (from a distance, and without modifying them, of course), it further normalises the subtle impression that systems should be immutable; that users should not modify their systems, and that they should be protected from themselves, relinquishing control of devices to an all-powerful vendor.
Individuals who grow up using immutable non-tinkerable systems suffer from greater tech illiteracy than those who grow up with user-tinkerable hackable systems.1 I find normalising the former concerning.
Snapshot-based rolling system updates
The following is my design for transactional system upgrades, based on experiments which took place mostly in 2022.
A key dependency is a filesystem which supports both copy-on-write and
subvolumes. I used btrfs, but zfs should be suitable too. Use a subvolume
for /, and separate subvolumes (or separate partitions) for the mount-points
which should never roll back (e.g., /home, /var/).
After installing the system, create a read-only snapshot of the root subvolume. This enables future rollbacks. Before installing a system upgrade, create a new copy-on-write subvolume, and then apply the upgrade. Essentially, the newly upgraded system is in a deduplicated pseudo-partition. Preferably, create a read-only snapshot too (to enable rolling back to exactly this point).
Note that subvolumes are mutable, so small changes can be made on top of each one, and snapshots exist as a mechanism to return to the exact state after an upgrade. This implies that it’s possible to install packages normally or experiment with the system freely, while having a rollback path.
Subvolumes are like branches: the live version of the system with ongoing changes. Snapshots are like tags: a frozen version of the system at a specific point. Of course, it’s trivial to create new snapshots from a subvolume, or new subvolumes from a snapshot.
Each time we create a new snapshot and subvolume (e.g., during a system upgrade), also create a UEFI bundle2, bundling the current kernel, initramfs and cmdline. The cmdline in particular hardcodes the snapshot or volume name. This bundle shows up in the UEFI boot picker, and is the path to boot into the system in an older state.
My specific implementation had two sets of bundles:
- A single “rolling” bundle with the latest kernel, pointing to a “live” subvolume.
- One bundle for each prior system upgrade (and its corresponding read-only snapshot) to boot into the system in a previous state.
The latter bundles are generated on each system upgrade, and the former is continuously overwritten.
When booting into a previous system state, / is in a read-only mode, and this
mode is mostly useful to create a new subvolume (effectively a mutable system
state branching off a known-working state), or to analyse and fix the “rolling”
subvolume. It’s also possible to simply use the system in this state with a
read-only root partition.
Whether to use snapshots, subvolumes, or both is a matter of policy.
Finally, tinkerers can opt to persist the current system state at any time, providing more breathing room for experiments with uncertain results.
Status of this work
I gradually abandoned this for two reasons:
- My hosts never had a “botched update” where I needed to roll back to their previous state. I only used the rolling-back feature when making sure that it worked, but never beyond that.
- The few times I messed up my system due to a configuration error, it still booted and I simply fixed my errors — I moved forwards rather than backwards.
If I had to maintain workstations or hosts for other users, I would pursue this further: the more users relying on it, the more motivation exists to further iterate.
For those wishing to study prior art on this topic, some of this is vaguely inspired by NixOS, although NixOS takes an entirely different approach to being able to boot from a previous system state, and its design is generally not portable to other distributions.
Blessing
Once a system has booted okay and has been confirmed working (automatically or by the user), they may opt to bless that subvolume. Blessing marks it as known-to-work. When low on disk space (e.g., the UEFI bundle won’t fit the ESP), older versions shall be automatically discarded, but at least one previous version is kept, and a blessed one shall be prioritised for retention.
Eventually, blessing could be done automatically (other approaches tend to do this, but not without their caveats).
Research on the topic classifies devices as “generative systems” (PCs and other systems which are open, tinker-friendly, and promote technical understanding) and “tethered appliances” (immutable devices meant only for passive consumption). The rise of the latter kind of systems leads to less technical literacy amongst their users. ↩︎
These are sometimes referred to as “UKI”, although a UKI is a UEFI bundle with a specific shape and set of properties — these are compatible but not strictly required for this design. ↩︎