Fusion of brainvisa-cmake, casa-distro and BrainVISA concepts for grouping software components

[sapetnioc]

Summary

I recently found out that brainvisa-cmake's group of components, casa-distro's "distro" and BrainVISA's Conda packages have many things in common and can be grouped in the same concept in order to be defined in a single place. Implementation of this fusion can have very limited impact on existing projects by just changing how brainvisa-cmake groups are defined in components_definition.py. But this will reinforce the interdependence between casa-distro and brainvisa-cmake since they both need to access to these software components groups. Therefore I open this discussion if anyone wants to comment this change I am working on.

The reason for the fusion

Definition of component groups in brainvisa-cmake is spread over all components definitions. This is not human friendly. It is both difficult to define a group and to know what is in a group and why. I decided to improve this by making a small change: define these groups in a dictionary. Some groups are defined as an addition of some components from another group. Therefore, to make definition of a group even more readable, I decided to add dependency between groups in the definition. If we add some metadata to a group definition (at least a short description should be added for brainvisa-cmake) we obtain the same structure used that casa-distro use to define the packages when creating Conda repositories. Moreover, some groups are conceptually identical to a corresponding "distro" in casa-distro. For instance brainvisa group is defined as "All the components included in BrainVISA release"; this is exactly the same concept we choose when we create a build environment for the distro brainvisa.

Therefore, it is possible centralize the definition of all these concepts. Centralization will make it much more easy to maintain the links between components, groups, distro and packages.

How will it work ?

There will be a single concept, called package, to define content of brainvisa-cmake groups of components, casa-distro distro and Conda packages. At first, packages will be defined in brainvisa-cmake next to the components in components_definition.py. This location may change in the future. The syntax will be the following:

packages = {
    {package-name}: {
        'about': {about}
        'alias': {alias names},
        'components': {components},
        'packages': {packages}
    }
}

{package-name} is a string containing the external package name. This will typically be used to create Conda packages and must be unique among all existing packages in known repositories. This is why many package names are prefixed with brainvisa- for instance (brainvisa-base, brainvisa-freesurfer, etc.). Internally (for instance in bv_maker or casa_distro commands), alternative names can be defined in {alias names} which is a string (for a single alias) or a list of strings.

{about} contains package metadata its content is based on the "about" section of Conda package metadata format. It should contain at least a summary entry.

{components} is a list of component names that are included in this package.

{packages} is a list of package names whose component are included (recursively) in this package.

An work in progress example on how package could be defined can be found in brainvisa-cmake conda branch.

What will it change ?

Probably nothing for a brainvisa-cmake or casa-distro user. Packages will be intepreted as standard component groups and alias mecanism will allow to be backward compatible with existing brainvisa-cmake's groups or casa-distro's distros.

However, this will enable to make changes in casa-distro to support two developpement environment types: singularity and conda.

What are the drawbacks ?

This fusion of concepts raises question about the versioning of the packages. Since brainvisa-cmake components will not be deployed directly anymore but always through a package definition, their versioning become useless on an external point of view. On the other hand, it is mandatory to set a version on the packages. To date, all the packages will have the version of the BrainVISA release they were built for. This is correspond to what we are currenlty doing with our monolithic container distribution.

Brainvisa-cmake and casa-distro are less independent. casa-distro is used to setup a development environment and create conda packages, it needs brainvisa-cmake to download sources the first time and to get packages information. On the other hand brainvisa-cmake behave differently depending on the type of build environment (singularity or conda). As a result, it is common to have to modify the two projects to fix a bug or implement a feature.

[Hboni]

Sounds great! This organization seems easy to understand. I was wondering about the package version. With this organization, may it be impossible to have a different package version at all. Even in a dev environment?
What I dislike about having the same version for all the packages is that in case of a version increase, if a package was not modified between versions, we may have different versions without modifications.