makefiles/docker.inc.mk: handle building in git worktree by cladmi · Pull Request #10303 · RIOT-OS/RIOT

cladmi · 2018-10-30T18:22:39Z

Contribution description

When building from a worktree, the common git directory was not mounted in docker.

This lead to the version not being set and issues with git-cache in
ubuntu bionic that could not execute the 'git hash-object' command.

Implementation details

I just test that .git is a file starting with gitdir to know it is in a worktree.
Any other solution is welcomed.

I then mounted the whole --git-common-dir in the same location.
Using the gitdir pointed by the .git file was not enough to make it valid.

I aslo tested overwriting the .git file with the .git repository but docker did not allow that.

Testing procedure

Original behavior untouched

In a normal repository, no .git directory is mounted when building with docker.

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all

# There should be no output with `-v path_to_dir/RIOT/.git:path_to_dir/RIOT/.git`

New behavior

This must be run from a git worktree (replace riot/master to the upstream master branch if different)

git worktree add ../git_test_worktree riot/master
cd ../git_test_worktree

Testing the RIOT_VERSION calculation from git

With riot/master we get an UNKNOWN version, (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20641     372   47684   68697   10c59 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: UNKNOWN (builddir: /data/riotbuild/riotbase))

With this PR the version is correctly detected (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all 
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20649     372   47684   68705   10c61 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: 2018.10-RC1-41-g72f71-pr/make/docker/handle_worktree)

Testing the behavior with `ubuntu:bionic` docker image

You must have an initialized git cache

./dist/tools/git/git-cache init

Build a docker image from RIOT-OS/riotdocker#42
I used docker build . -t riot/bionic from the repository.

Compiling a package using the bionic image fails in master as git hash-object tries to detect the current directory as a git directory.

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:487: recipe for target 'pkg-prepare' failed
make: [pkg-prepare] Error 2 (ignored)
rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:491: recipe for target 'pkg-build-littlefs' failed
make: *** [pkg-build-littlefs] Error 2
/home/harter/work/git/worktree/riot_master/makefiles/docker.inc.mk:100: recipe for target '..in-docker-container' failed
make: *** [..in-docker-container] Error 2
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

It works with this PR

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
touch /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded
"make" -C /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs -f /data/riotbuild/riotbase/pkg/littlefs/Makefile.littlefs
   text    data     bss     dec     hex filename
  87035    1180   49336  137551   2194f /data/riotbuild/riotproject/examples/filesystem/bin/native/filesystem.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

Issues/PRs references

The RIOT_VERSION issue was detected in #9645 (comment) and the git-cache issue was detected when running Release compilation with bionic image RIOT-OS/Release-Specs#76 (comment)

When building from a worktree, the common git directory was not mounted in docker. This lead to the version not being set and issues with git-cache in ubuntu bionic that could not execute the 'git hash-object' command.

cladmi · 2018-10-30T18:26:44Z

A test in mac would be great too if you could @smlng

cladmi · 2018-10-30T18:33:56Z

I could test without it for the release (or just use the patch in my branch), but having the release tag working with the upcoming bionic images could be a plus. Up to you @jia200x

jia200x · 2018-10-30T19:09:32Z

I could test without it for the release (or just use the patch in my branch), but having the release tag working with the upcoming bionic images could be a plus. Up to you @jia200x

I agree. Let's get this in

makefiles/docker.inc.mk

jcarrano

Tested. Works.

cladmi · 2018-10-31T14:57:01Z

Thanks for the review.

cladmi · 2018-10-31T14:58:28Z

Backport provided in #10307

makefiles/docker.inc.mk: handle building in git worktree

61a3e5d

When building from a worktree, the common git directory was not mounted in docker. This lead to the version not being set and issues with git-cache in ubuntu bionic that could not execute the 'git hash-object' command.

cladmi requested review from bergzand, jcarrano, jnohlgard and smlng October 30, 2018 18:23

cladmi mentioned this pull request Oct 30, 2018

Update to Ubuntu Bionic RIOT-OS/riotdocker#42

Merged

cladmi added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Oct 30, 2018

jia200x added the Process: needs backport Integration Process: The PR is required to be backported to a release or feature branch label Oct 30, 2018

cladmi mentioned this pull request Oct 30, 2018

Different build behavior between murdock and riot/riotbuild:latest image #9645

Closed

6 tasks

jcarrano reviewed Oct 31, 2018

View reviewed changes

makefiles/docker.inc.mk Show resolved Hide resolved

jcarrano approved these changes Oct 31, 2018

View reviewed changes

jcarrano merged commit 77dc923 into RIOT-OS:master Oct 31, 2018

cladmi mentioned this pull request Oct 31, 2018

makefiles/docker.inc.mk: handle building in git worktree [backport 2018.10] #10307

Merged

cladmi added this to the Release 2018.10 milestone Nov 1, 2018

jia200x modified the milestone: Release 2018.10 Nov 1, 2018

cladmi deleted the pr/make/docker/handle_worktree branch November 2, 2018 11:08

jcarrano mentioned this pull request Dec 5, 2018

make: fix BUILD_IN_DOCKER when RIOT is submodule #10550

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

makefiles/docker.inc.mk: handle building in git worktree#10303

makefiles/docker.inc.mk: handle building in git worktree#10303
jcarrano merged 1 commit intoRIOT-OS:masterfrom
cladmi:pr/make/docker/handle_worktree

cladmi commented Oct 30, 2018

Uh oh!

cladmi commented Oct 30, 2018

Uh oh!

cladmi commented Oct 30, 2018

Uh oh!

jia200x commented Oct 30, 2018

Uh oh!

Uh oh!

jcarrano left a comment

Uh oh!

cladmi commented Oct 31, 2018

Uh oh!

cladmi commented Oct 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cladmi commented Oct 30, 2018

Contribution description

Implementation details

Testing procedure

Original behavior untouched

New behavior

Testing the RIOT_VERSION calculation from git

Testing the behavior with ubuntu:bionic docker image

Issues/PRs references

Uh oh!

cladmi commented Oct 30, 2018

Uh oh!

cladmi commented Oct 30, 2018

Uh oh!

jia200x commented Oct 30, 2018

Uh oh!

Uh oh!

jcarrano left a comment

Choose a reason for hiding this comment

Uh oh!

cladmi commented Oct 31, 2018

Uh oh!

cladmi commented Oct 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Testing the behavior with `ubuntu:bionic` docker image