Switch /etc to use confext in mutable mode

[pothos]

• Make systemd-confext automatically use the mutable mode

  So far we had a custom overlay mount for /etc that provided the A/B
  updated files from /usr in a lowerdir. Since then we upstreamed a
  mutable mode for sysext and confext.
  We can now switch over to it and provide a default confext by using the
  mutable mode. For user-provided to be supported we need to wait for the
  atomic remount to be there so that a reload is less impacting. Also, one
  has to be careful with the stacking order for user-provided extensions
  and modes other than mutable. The read-only mode also needs more design
  work in Flatcar.

• Drop old ensure-sysext.service workaround

  When systemd-sysext didn't support issuing a daemon reload, we had this
  service do it. This is now doing the same thing again while not needed.
  The service also added support to using .wants but in general .upholds
  should be used. For extensions that use .wants we can still start their
  services when we set the extension up from the initrd. Left is just the
  case where the extension is loaded live but since this was a
  Flatcar-specific workaround it's not something we want to support and
  users either should migrate to .upholds, start the service manually (or
  with the target restart as done in this workaround), or do a reboot when
  they add new extensions live. We also plan to add a service restart
  field to the extension-release metadata file so that one doesn't need to
  use .upholds and it even covers more cases.

• systemd: Define order for confext and sysext

  For a confext to influence how things shipped in a sysext behave or even
  how sysext itself behaves it's good to define that confext runs first.
  This is also explicitly done in bootengine. With the skip logic we
  ideally don't set up extensions again during boot but that only works if
  the contents can be ensured to be equal and the fallback is a full
  refresh otherwise.

These changes are needed to ship the default /etc files as confext in the mutable mode.

How to use

With the scripts PR that uses it.

Testing done

See scripts PR

[chewi]

Looks good!

I'm really happy about the noop thing. I hated that we needed to expose that kludge to bakery users. It would be even better if sysupdate just permitted no config, but this will do.

Regarding prepare-mutable.conf, is there anything to ensure that /var gets mounted before systemd-confext.service starts? Do we maybe need a RequiresMountsFor? Or perhaps even a BindsTo?

[pothos]

Regarding prepare-mutable.conf, is there anything to ensure that /var gets mounted before systemd-confext.service starts? Do we maybe need a RequiresMountsFor? Or perhaps even a BindsTo?

Good question, in general it sounds good to do this but I've also seen that the zfs sysext has this drop-in to let sysext run before udev which I don't really understand yet. Probably zfs users won't store extension images on a zfs /var but encoding that sysext requires /var could cause them troubles?

I guess it's also important to say that while we have a test case for a separate /var in kola, this setup is not that well supported and ideally the /var mount would be done from the initrd so that tmpfiles and similar have a proper chance to run.
So best is maybe to not encode a dependency for now but document that one might want to look into this when setting up a separate /var.