Add the ability to install additional soft support dependencies at runtime

[gilesknap]

in epics-containers we prefer to have a small number of reusable generic IOCs instead of rebuilding a container image for every combination of support modules that legacy IOCs currently contain.

Most beamlines have a beamline support module which simply adds a number of custom EPICS DB templates. These are included into many of the beamlines IOC's dependencies.

There are also many other support modules that only add DB.

To accommodate these we propose a mechanism by which each ioc instance can declare additional dependencies in it's config folder. Perhaps in the form of a additional_support.yaml file that lists support modules.

The feature would interpret the additional support list and clone the support into the ALREADY EXISTING (just never used before) data PVC that each ioc instance is allocated. It would also obtain the module.ibek.support.yaml file and any other ibek-support files needed such as pvi files (a challenge here is that this should be from the ioc-xxx submodule version of ibek-support).

symlinks and paths would be set up so that the addtional support is transparently available to ibek as it generates runtime assets declared in ioc.yaml.

[gilesknap]

After a conversation with @coretl we have the following proposal for adding additional soft support at runtime.

1. In the config folder have additional yaml files called eg. extras01.yaml
2. the contents would look something like this:-

   # yaml-language-server: $schema=https://gitlab.diamond.ac.uk/controls/support/i04-support...schema.json    
   
   description: i04 database templates
   source: https://gitlab.diamond.ac.uk/controls/support/i04-support
   version: R3-3
   
   entities:
     - type: i04-support.some_template
       P: BL04I-something
       etc.

3. The support module may have an additional folder /etc/ibek-support which will have contents like ibek-support/XXX i.e.
   • module_name.ibek.support.yaml: describes the entity models for entities in the above yamlexample yaml above.
   • template_name.pvi.template: templates for PVI definitions
4. There will also be /etc/module_name.ibek.schema.json: the schema file generated from module_name.ibek.support.yaml and referred to at the top of the extras yaml
5. Note that these additional files are optional but if not supplied then no entities can be specified in extrasNN.yaml and instead some hand coded ioc.db must be supplied in the /config folder.
6. Additional NOTE: we will automatically process the following two special file names and include the necessary dbloadrecords in st.cmd:
   • ioc.db
   • ioc.subst

When start.sh encounters an extras file it will:-

• Clone the referenced repo into /data/support/module_name
  • note that /data is a pre-existing PVC that all IOCs have
• symlink /data/support/module_name/etc/ibek-support/module_name.ibek.support.yaml -> /data/ibek-defs
• call ibek runtime generate as before except that the support yamls come from both /epics/ibek-defs/* and /data/ibek-defs/* and that the ioc.yaml has the extras*.yaml entities appended.
• finally also make sure that the (source) Db folders in /data/support/ are included in the DB path when calling MSI.

The idea here is that we recreate a very similar folder structure under /data as to that under /epics and that ibek now consumes the files from both.

The other important idea is that versioning is by support module - so the support yaml and schema come from the same version of the support module and there is no intermediate ibek-support submodule to go through. This requires that these support modules have some CI that generates schema and publishes it as an artifact.

Finally there it is possible that we will start using the PVC for mounting the config folder instead of a config map. This work should be investigated first to see how well it aligns with the above.

TODO: I have a sneaky feeling that having a PVC per IOC is counter productive and we should have a single pvc where each IOC mounts a subfolder instead. This aligns with what we do for opis, autosave and runtime folders. @coretl do you have thoughts on this?

[coretl]

4. There will also be /etc/module_name.ibek.schema.json: the schema file generated from module_name.ibek.support.yaml and referred to at the top of the extras yaml

I think we discussed generating these as artifacts in CI rather than putting them in the module.

• ioc.subst

All our existing things are called something.substitutions, do we want to change this?

When start.sh encounters an extras file it will:-

If there is something there already it should probably update in place, and as long as it is already checked out then an update failure should be converted to a warning

TODO: I have a sneaky feeling that having a PVC per IOC is counter productive and we should have a single pvc where each IOC mounts a subfolder instead. This aligns with what we do for opis, autosave and runtime folders.

One PVC for everything sounds fine