`oldkern` no longer a safe fallback after updating kernel

[danielwe]

Distribution (run cat /etc/os-release):

NAME="Pop!_OS"
VERSION="22.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 22.04 LTS"
VERSION_ID="22.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
LOGO=distributor-logo-pop-os

Related Application and/or Package Version (run apt policy $PACKAGE NAME):

kernelstub:
  Installed: 3.1.4~1674849051~22.04~851230b
  Candidate: 3.1.4~1674849051~22.04~851230b
  Version table:
 *** 3.1.4~1674849051~22.04~851230b 1001
       1001 http://apt.pop-os.org/release jammy/main amd64 Packages
       1001 http://apt.pop-os.org/release jammy/main i386 Packages
        100 /var/lib/dpkg/status

Issue/Bug Description:

Given the heuristics for determining the current and previous images, a broken kernel may end up in the oldkern image. This means it's possible to end up with both current and oldkern being broken if two successive kernel releases are broken on your hardware.

Steps to reproduce (if you know):

• Boot into oldkern image.
• Run uname -a and take note of kernel version
• Update kernel packages. If you're lucky, sudo apt upgrade will do. If you already had the newest kernel installed such that this does not update linux-image-generic, there are a few options:
  • Use mainline to install a newer kernel
  • Add a kernel staging branch and update packages, for example:

  $ sudo apt-manage add popdev:linux-6.5.6
  $ sudo apt update
  $ sudo apt upgrade
  

• Try to reboot into oldkern image. If successful, run uname -a and observe that it is different from before. Specifically, it's the version that was in the current image before the kernel upgrade.
• Ponder the consequences of this if the reason you booted into oldkern in the first place were that current was broken for your hardware. Now, that broken kernel has been moved to oldkern and your reboot would have failed.
• What if the kernel you just installed is also broken on your hardware? Now both bootloader entries point to broken images.

Expected behavior:

I would prefer if oldkern were a safe fallback that always points to an image that is known to boot successfully. In particular, I suggest the following heuristic:

• If the image that's being put into current is different from the currently running image, put the currently running image in oldkern
• Otherwise, don't touch oldkern at all

That way, the oldkern image will never be changed to a version that isn't proven to be bootable.

This is relevant because kernels that fail to boot on common hardware are still being released from time to time, see pop-os/pop#3152.

The "currently running" heuristic gets complicated when chrooting from a live USB or similar. Perhaps it's possible to instead identify which images in /boot have ever booted successfully? Or even better, have executed a successful shutdown.

Other Notes:

Related: #40, which anticipated this issue.