kernel updates require two reboots

[janxkoci]

What Happened?

For some reason - and unique in the Linux world - the new versions of Appcenter require two reboots when there are kernel updates in the queue. This is highly unusual.

Steps to Reproduce

1. Open Appcenter to see available updates.
2. If you see the linux kernel under "System updates" - bingo!
3. Hit "Update all" - note that "System updates" has a "Download" button on the right side, rather than "Update".
4. The Appcenter will prompt you to reboot - do it.
5. After a reboot, the Appenter will again show a notification that updates are available.
6. Open Appcenter and see that "System updates" are still there and the linux kernel is still among them.
7. Hit update (again).
8. You will be (again) prompted to reboot. Again, do it.
9. After the second reboot there are finally no more updates.

Expected Behavior

Kernel updates have always required only one reboot. It's a nice tradition to keep.

OS Version

7.x (Horus)

Software Version

Latest release (I have run all updates)

Log Output

No response

Hardware Info

No response

[janxkoci]

Actually, now it seems that every update requires a reboot - the notification comes up even after updating a single app from flathub.

[janxkoci]

updating with apt does not require double-reboot

Today, AppCenter crashed while updating (I've seen both linux kernel and also appcenter in the list of updates, the second could be responsible for the crash), so I decided to try updating from CLI (first flatpak update, then sudo apt update && sudo apt upgrade). Later, I rebooted my laptop and opened AppCenter. It did not bother me about second reboot!

So this is clearly something AppCenter does on its own for some reason. One major problem with the current behaviour is that AppCenter holds all system updates until reboot when kernel is also in the update list! (see e.g. elementary/files#1858 (comment))

speculation

One reason could be that AppCenter cannot update GRUB? I always see some progress bar during reboots after kernel updates, with a short reboot at the end of the progress, before the machine starts to boot into my system. Is this somehow equivalent to what apt does at the end of kernel updates (updating GRUB and such)? Can AppCenter also do it then, rather than during next reboot?

[danirabbit]

This isn't an AppCenter specific issue. This is because updates are installed offline and affects pretty much all offline update mechanisms on Linux.

Here's a relevant issue report in PackageKit: PackageKit/PackageKit#510

There's been some progress recently upstream about resolving this, I believe in systemd. I can't find the information at the moment but this is an underlying Linux architecture issue and not something that can be solved in AppCenter

[janxkoci]

To clarify: this issue is about the current version of AppCenter included in OS 7 behaving differently then older versions of AppCenter, as well as all update mechanisms I know from Ubuntu in the past 10+ years. It all works as I'm used to, except the AppCenter in OS 7, which does things differently.

PS: in the terminology of the linked issue, elementary OS 7 does three reboots to apply a kernel update. So there is still one more than there needs to be. I didn't count the short reboot after the progress bar as a real reboot in my original post. Actually, there may be four this way, not sure.

PPS: One more example to illustrate - on my second laptop with Ubuntu MATE 22.04 (i.e. the same version as the base of elementary OS 7), applying kernel updates takes exactly one (visible) reboot, no matter if I use apt or the GUI updater. Non-kernel updates are applied directly and closing and reopening updated app will open its newer version. Same for scientific package managers like conda I use at work - running conda update r-base will give me the latest version of R the next time I run its command. On the other hand, kernel stays the same until the next reboot, when the system boots into the new kernel (without any noticeable delays or semi-reboots, afaict). In contrast, the AppCenter in elementary OS 7 holds all system updates (apt-based, I assume) until next several rounds of reboots, anytime there is also a kernel update included in the list. This is a vastly different experience from any Ubuntu-based distro I used in the past, including older versions of elementary OS. I understand you are innovating here, but this one noticeably hurts.

[danirabbit]

@janxkoci Yes, to clarify: AppCenter now installs all system updates offline so the behavior will be different than installing updates via apt while online. Offline updates require an extra reboot but the trade off is a more stable updates process with a lower chance of failure

[janxkoci]

@danirabbit Ok, interesting, but the link still only talks about two reboots, while I experience 3-4 reboots (I need to count them properly next time using their definition).

Also, online updates seem to be rock-solid - the same Ubuntu MATE laptop has only 32gb eMMC storage and with 9gb in Dropbox and a few apps I usually have only some 1.2 - 1.5gb of free disk space (below 1gb Dropbox stops syncing). I often observe that updates - especially kernel updates - regularly run out of space in the middle (as in df -h reporting 0B left on main partition), but apt still manages to unpack everything and apply the updates. It's quite impressive tbh. (I usually autoremove and autoclean right after that.)

[danirabbit]

Ah here we go, here's a talk about Soft Reboot: https://www.youtube.com/watch?v=5HO1PAcsM7g It looks like this landed in Systemd in July, so maybe in OS 8 we'll be able to have only a single reboot when installing updates

[janxkoci]

Ok, I don't think that's the same issue. My issue is that I have to reboot twice all the way to the desktop to apply system updates, while the video and the linked PackageKit issue only talk about the short reboots during booting sequence (i.e. way before I even see login screen).

I also noticed a reboot is sometimes required even without a kernel update. One such case was the five days ago, this is the log entry for 20th of September:

Start-Date: 2023-09-20  19:23:22
Commandline: packagekit role='update-packages'
Upgrade: appstream-data-pantheon-icons:amd64 (5.0+r292~ubuntu7.1, 5.0+r293~ubuntu7.1.1), appstream-data-pantheon:amd64 (5.0+r292~ubuntu7.1, 5.0+r293~ubuntu7.1.1), base-files:amd64 (12ubuntu4.4+elementary11~ubuntu22.04.1, 12ubuntu4.4+elementary12~ubuntu7.1), appstream-data-pantheon-icons-hidpi:amd64 (5.0+r292~ubuntu7.1, 5.0+r293~ubuntu7.1.1)
End-Date: 2023-09-20  19:23:25

I don't see anything obvious that should require a reboot there (only one reboot to desktop was needed though).

So to recap:

1. Too many reboots are required to apply system updates,
   a. partly due to switching to offline updates,
   b. and partly due to a bug in elementary OS and/or AppCenter.
2. Offline updates are the future, so I should get used to rebooting nearly every day.
3. If I cannot be bothered to reboot every day (I'm not on Windows after all, my average uptime is normally a week or two), I can ignore AppCenter and use CLI to perform all updates.

Did I miss anything?

[danirabbit]

Yes, all system updates regardless of whether or not they contain kernel updates require a restart because they are installed offline. The only updates that install in the session are Flatpak updates.

We only push out a set of updates about once a month, unless you're running early access, so you should plan to restart your system once a month to stay up to date.

If you're saying you're rebooting all the way into the desktop multiple times then something there is wrong because what you should see is a reboot, installation of updates happening offline, another reboot, and then your updated desktop

[janxkoci]

Well I see system updates definitely more often than once a month, but some of those will be Ubuntu updates.

If you're saying you're rebooting all the way into the desktop multiple times then something there is wrong because what you should see is a reboot, installation of updates happening offline, another reboot, and then your updated desktop

And that's exactly what this issue is about. PS: see also "steps to reproduce" in the first post.