System tests fail by definition for build-time changes in adapters/solvers

[MakisH]

Context

We currently have all the Dockerfiles to create adapter images in the system tests repository.

Let's take the CalculiX adapter for example. It's Dockerfile has the following structure:

# Base preCICE image (for specific distribution and preCICE branch)
ARG from=precice/precice-ubuntu1604.home-develop
# Base solver/dependencies image
ARG base_solver=precice/calculix

# ... get / copy dependencies ...

# Get CalculiX source and the adapter FOR THE SPECIFIC CALCULIX VERSION
WORKDIR /home/precice
RUN curl -s http://www.dhondt.de/ccx_2.15.src.tar.bz2 | tar -xj
ARG branch=develop
RUN git clone --branch $branch https://github.com/precice/calculix-adapter.git

# ... build the adapter (adapted CalculiX) ...

Problem

With this structure, every time we update the default version of a solver, we automatically get a system tests failure, which we need to fix with a different PR. We often get the situtation that we cannot test an update before we implement it and break the tests.

Suggested solution

Let's move the solver and adapter Dockerfiles in the respective adapter repositories. The system tests then would need to build the images based on the Dockerfiles that come with the solver. @Eder-K is this technically possible?

Advantages

• Stop breaking the system tests for known compile-time changes
• Keep the building scripts together with the code: easier to (remeber to) maintain
• Dockerfiles would be a useful documentation source for users and they would be together with the code

Disadvantages

• We don't enforce testing specific solver versions
• It solves only part of the problem, as we still need to update the tutorials repository separately (if applicable).

One could even see this as a feature (alarm for breaking changes), but I would argue that we should not be alarmed for changes that are known and intended. Besides, the system tests will still fail if more changes are required (e.g. in the configuration files (see OpenFOAM adapter) or name of solvers (see deal.ii adapter)).