Updates and upgrades of containers

[evgeni]

This sort of a super-set of #66, focusing not purely on PostgreSQL but more on the questions "how to detect an update/upgrade is needed" and "which steps should be part of the update/upgrade".

Types / terminology

• Update - "same stream" update, so e.g. 3.16.0 to 3.16.1 (or, rather, "to latest 3.16.z")
• Upgrade - "stream change" upgrade, so e.g. from 3.15.1 to 3.16.0

Current (package based) workflow

• stop services
• upgrade packages
• run installer (which starts services)

This is masked by foreman-maintain in some scenarios, but the flow as a whole remains and is identical for updates and upgrades.

For disconnected installations there is an additional step of configuring the new offline repositories.

Possible container-based workflows

1. the naïve copy

Basically the same as the package based flow

• stop services
• obtain a the updated version of foremanctl (if any, esp important for upgrades that change the container stream)
• run deploy (which will pull containers and start everything)

Disconnected users will have to "pre-pull" the containers into the local setup.

"stop services" is a command we don't have today. we could implement a role to do so and then expose a foremanctl update that does the stop and then a deploy?

2. the podman auto-update

podman can automatically update containers which we could use, but:

• it inspects the running container only, which means if we change the on-disk .container file to point to a different stream it won't trigger (but we don't strictly need that, we know the change event happened and can trigger the service restart ourself)
• to make it work for disconnected, we'd need to use AutoUpdate=local and rely on the local images being updated "somehow" (user or us pulling from the registry)
• at least pulp needs migrations to run "manually" (not as part of the container start like foreman and candlepin)

3. something completely different

I am sure we can come up with something else. The probably most important thing is to have a trigger to start upgrade actions (stopped services is one, changed container definitions is another, etc)