Updating Arch Linux

Created 2024-10-09

I run Arch Linux (by the way) on my desktop, laptop, and a couple of personal servers. As a rolling release distro, Arch does not have the same guarantee of package capability that a traditional distro like Ubuntu offers (that said, I always end up installing many apps from PPAs and other out-of-band means on Ubuntu, so I'm not sure how much that guarantee means). I've had very few problems updating in the 10 years or so I've run Arch as a daily driver, but there's always the possibility of something going horribly wrong. I thought it might be worth presenting the solution I've developed in case it might be interesting or helpful to others.

The foundation of this approach is LVM, though it would also work similarly on other file systems with copy-on-write snapshots like ZFS or btrfs. When I provisioned the systems, I allocated most of the space to a root logical volume, but left 10-20Gb of free space for snapshots.

Before I update, I take a snapshot of the root volume. Then I perform the update, and if anything goes wrong I can revert back to the previous state. I have had to do this a handful of times over the years, most recently on my cloud VPS when a headscale (self-hosted tailscale control server) update broke its internal database.

To avoid immediately filling up the snapshot volume with the downloaded package data, I actually download (but not install) the packages before taking the snapshot. On Arch this looks like pacman -Syuw.

The snapshot is named with the current date, to shame me if I go too long between updates.

The full script is pasted below. I vary the snapshot size on different machines depending on how much free space I have in LVM.

update.sh

#!/bin/bash
echo "Downloading files..."
sudo pacman -Syuw
OLD=$(ls /dev/volgroup/ | grep update | tail -n 1)
if [ "$OLD" != "" ]; then
   echo "Most recent snapshot: $OLD"
   read -p "Delete snapshot? (y/n) " DELSNAP
   if [ "$DELSNAP" = "y" ]; then
       sudo lvremove /dev/volgroup/$OLD
   fi
fi
NEW=$(date +"update%Y%m%d")
echo "Creating snapshot /dev/volgroup/$NEW..."
sudo lvcreate --size 3500M --snapshot --name $NEW /dev/volgroup/root
read -p "Ready to update? (y/n) " READY
if [ "$READY" = "y" ]; then
    sudo pacman -Su
fi
echo "Done."

Reverting Snapshots

In the unlikely event of an emergency, I can revert the snapshot. This is a somewhat delicate process, because changing the entire filesystem out from under a running operating system would be a bad idea. Not that it's even possible - I'm pretty sure LVM will refuse to merge a snapshot into a mounted volume.

The first step is to boot into another operating system, from a USB drive if it's a computer I can touch, or by mounting an iso file on my cloud VPS though the provider's interface.

After getting past any LUKS encryption (on all of my machines except the cloud server), the following command will revert the root volume to the state captured by the snapshot:

lvconvert --merge /dev/volgroup/update20241009

The danger in this approach is that the snapshot volume will fill up if the update changes too many files. A full snapshot will no longer track changes and will not be able to be merged. Often by the time I get around to updating my previous snapshot will be full, though it's not an issue since by then I'm long past the need to revert. To check the status of a snapshot, run sudo lvs. The snapshot will be listed alongside your other logical volumes.

I'm not sure when I first wrote my update script, but it has served me well. Automating the process ensures I will always have have the snapshot as a backup. I've used it for updates outside of pacman as well - I've had to revert snapshots after a Nextcloud docker container update gets stuck (one of the reasons I periodically consider ditching it...that's probably another post).