Feature/ci run boardfarm by rmelotte · Pull Request #1492 · prplfoundation/prplMesh

rmelotte · 2020-06-29T06:59:45Z

Run boardfarm using docker-only in CI, until the docker-compose integration can be done.

arnout · 2020-06-29T16:36:45Z

tools/docker/boardfarm-ci/Dockerfile

+RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+    docker-ce \
+    docker-ce-cli \
+    containerd.io \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
+RUN chmod 755 /usr/local/bin/docker-compose


For my info: why isn't all this simply taken from Debian/Ubuntu repos? Are we using such advanced features?

Good question, I took the dockerfile from #1483 and tried to make only the changes I needed.
Maybe @odkq needs a more recent version than the base image provides?

We could, debian stable is 1.21 and I don't think we are going to use later development, but as documentation is not tagged (always the latest) with this we have the mechanism in place for future upgrades (just change the 1.26.0 in the url and test)

arnout · 2020-06-29T16:39:41Z

tools/docker/boardfarm-ci/Dockerfile

+COPY requirements.txt /app/requirements.txt
+
+WORKDIR app
+RUN pip3 install -r requirements.txt


Why do we need this? Before, we just took the requirements.txt from boardfarm. I.e.

RUN git clone https://github.com/mattsm/boardfarm.git \ && cd boardfarm \ && git checkout 100521fde1fb67536682cafecc2f91a6e2e8a6f8 \ && pip3 install -r requirements.txt && python3 setup.py install

Yes, it does mean that all the pip stuff has to be re-downloaded whenever we change the boardfarm hash, but that's going to be rare anyway.

Boardfarm itself is not installed in this setup, it is used from the rootdir/boardfarm dir directly to allow for any changes made locally to be used.

As we do not expect to change boardfarm itself, this will be changed to installing as before, cloning and the rest.

@arnout good point, I hadn't noticed boardfarm itself also included the requirements, I though it was added by us.

@odkq with these changes, boardfarm is installed in the image itself. Is it something you want to do differently?

arnout · 2020-06-29T16:41:24Z

tests/run_bf.sh

 export PYTHONPATH

-exec python3 "${bf_dir}"/bft -c "${bf_plugins_dir}"/boardfarm_prplmesh/prplmesh_config.json -n prplmesh_docker -x test_flows
+exec bft -c "${bf_plugins_dir}"/boardfarm_prplmesh/prplmesh_config.json -n prplmesh_docker -x test_flows


When Oleksii originally integrated boardfarm, I asked not do to this exactly because it requires installing boardfarm, which I wanted to avoid :-)

But I guess it's OK now that we have it in docker.

This change is actually bogus; I added a script run_bf_compose.sh that still uses the boardfarm in it's local copy, as above

.gitlab-ci.yml

rmelotte · 2020-06-30T13:56:13Z

Some of the commits belongs to @odkq , so I still need to add a SOB line for them.

arnout

A few of my comments haven't been addressed yet, but it's anyway stuff that can be done later, so approving already.

abelog

In general, LGTM.
Small questions left, but nothing big.

.gitlab-ci.yml

To run the boardfarm tests, we need an image containing boardfarm. Add an image that contains all the boardfarm dependencies, fetch a specific commit (the latest one from the master branch) and install it. The dockerfile was adapted from the existing work on: #1483 Also add the image to .gitlab-ci to get it built automatically. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

To use an already-installed boardfarm version we don't need to know it's path, we can assume it's already in PATH. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

Because prplMesh is not currently relocatable, we have to run it both in the same directory it was built in, but also as the same user. It is currently built with the root user inside a docker container, with the USER variable set as ''. As a consequence, we have to run it as root as well, and explicitly set USER to an empty string (because boardfarm relies on it's presence). Since we want to run boardfarm in docker and boardfarm starts containers itself, we have to use a docker image where docker itself is available from within the container. For this 3 options were possible: - The current one, running another docker daemon inside the first docker container. It is a little bit overkill, and makes it hard to preserve the cache between runs. - Run with a Docker executor and bind-mount the Docker socket. It requires some care when we want to mount volumes from within a container, because the volumes have to be available on the host. - Run with a shell executor. This approach has the benefit of always having the prplMesh tree cloned and the artifacts available on the host. Since we currently run the containers as root however, the Gitlab runner won't be able to clean up files in-between runs. The option 2 was chosen, because it was the easiest to use at the moment. For now the job also needs only the docker builds, so add a "need" for it. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

When a command failed to run, raise an exception early, and add the stdout and stderr output of the command in the message. These changes were taken and adapted from: #1483 Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

With #1486, "name" was changed to "docker_name". The name in "check_status" was not updated, so do it now. It fixes the following error: Error: No such container: controller See for example: https://gitlab.com/prpl-foundation/prplMesh/-/jobs/625370585 Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

With #1486, the first device to start has been changed to be the agent. Since the agent cannot be operational unless there is a controller already running on the same network, something else has to be used for "check_status". For now, use netcat to check that "device_get_info" works. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

rmelotte added this to the M3 - Workable product milestone Jun 29, 2020

rmelotte self-assigned this Jun 29, 2020

rmelotte force-pushed the feature/ci-run-boardfarm branch 3 times, most recently from 4bb57c7 to 72e6079 Compare June 29, 2020 16:11

arnout reviewed Jun 29, 2020

View reviewed changes

rmelotte force-pushed the feature/ci-run-boardfarm branch 2 times, most recently from cff6b21 to 39b1296 Compare June 30, 2020 13:30

rmelotte marked this pull request as ready for review June 30, 2020 13:30

rmelotte requested review from kantera800, morantr, tomereli and vitalybu as code owners June 30, 2020 13:30

arnout approved these changes Jul 1, 2020

View reviewed changes

abelog approved these changes Jul 3, 2020

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

.gitlab-ci.yml Show resolved Hide resolved

aciliketcap approved these changes Jul 3, 2020

View reviewed changes

rmelotte force-pushed the feature/ci-run-boardfarm branch from 39b1296 to 690497b Compare July 6, 2020 07:55

rmelotte added the ready for merge PR is ready to be merged (automatically) label Jul 6, 2020

arnout force-pushed the feature/ci-run-boardfarm branch 2 times, most recently from 2beda19 to 3b7eebf Compare July 6, 2020 09:01

rmelotte added 4 commits July 6, 2020 12:22

tests: boardfarm: use the installed version of boardfarm

3fd1f05

To use an already-installed boardfarm version we don't need to know it's path, we can assume it's already in PATH. Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>

arnout force-pushed the feature/ci-run-boardfarm branch from 3b7eebf to 84d0507 Compare July 6, 2020 12:22

rmelotte added 2 commits July 6, 2020 16:31

mergify bot merged commit cdb4ff1 into master Jul 6, 2020

mergify bot deleted the feature/ci-run-boardfarm branch July 6, 2020 17:38

mergify bot removed the ready for merge PR is ready to be merged (automatically) label Jul 6, 2020

Conversation

rmelotte commented Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rmelotte commented Jun 30, 2020

Uh oh!

arnout left a comment

Choose a reason for hiding this comment

Uh oh!

abelog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rmelotte commented Jun 29, 2020 •

edited

Loading