Skip to content

makefiles/docker.inc.mk: handle building in git worktree [backport 2018.10]#10307

Merged
cladmi merged 2 commits intoRIOT-OS:2018.10-branchfrom
cladmi:backport/2018.10/pr/make/docker/handle_worktree
Nov 2, 2018
Merged

makefiles/docker.inc.mk: handle building in git worktree [backport 2018.10]#10307
cladmi merged 2 commits intoRIOT-OS:2018.10-branchfrom
cladmi:backport/2018.10/pr/make/docker/handle_worktree

Conversation

@cladmi
Copy link
Contributor

@cladmi cladmi commented Oct 31, 2018

Backport of #10303

Contribution description

When building from a worktree, the common git directory was not mounted in docker.

This lead to the version not being set and issues with git-cache in
ubuntu bionic that could not execute the 'git hash-object' command.

Implementation details

I just test that .git is a file starting with gitdir to know it is in a worktree.
Any other solution is welcomed.

I then mounted the whole --git-common-dir in the same location.
Using the gitdir pointed by the .git file was not enough to make it valid.

I aslo tested overwriting the .git file with the .git repository but docker did not allow that.

Testing procedure

Original behavior untouched

In a normal repository, no .git directory is mounted when building with docker.

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all

# There should be no output with `-v path_to_dir/RIOT/.git:path_to_dir/RIOT/.git`

New behavior

This must be run from a git worktree (replace riot/master to the upstream master branch if different)

git worktree add ../git_test_worktree riot/master
cd ../git_test_worktree
Testing the RIOT_VERSION calculation from git

With riot/master we get an UNKNOWN version, (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20641     372   47684   68697   10c59 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: UNKNOWN (builddir: /data/riotbuild/riotbase))

With this PR the version is correctly detected (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all 
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20649     372   47684   68705   10c61 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: 2018.10-RC1-41-g72f71-pr/make/docker/handle_worktree)
Testing the behavior with ubuntu:bionic docker image

You must have an initialized git cache

./dist/tools/git/git-cache init

Build a docker image from RIOT-OS/riotdocker#42
I used docker build . -t riot/bionic from the repository.

Compiling a package using the bionic image fails in master as git hash-object tries to detect the current directory as a git directory.

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:487: recipe for target 'pkg-prepare' failed
make: [pkg-prepare] Error 2 (ignored)
rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:491: recipe for target 'pkg-build-littlefs' failed
make: *** [pkg-build-littlefs] Error 2
/home/harter/work/git/worktree/riot_master/makefiles/docker.inc.mk:100: recipe for target '..in-docker-container' failed
make: *** [..in-docker-container] Error 2
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

It works with this PR

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
touch /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded
"make" -C /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs -f /data/riotbuild/riotbase/pkg/littlefs/Makefile.littlefs
   text    data     bss     dec     hex filename
  87035    1180   49336  137551   2194f /data/riotbuild/riotproject/examples/filesystem/bin/native/filesystem.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

Issues/PRs references

The RIOT_VERSION issue was detected in #9645 (comment) and the git-cache issue was detected when running Release compilation with bionic image RIOT-OS/Release-Specs#76 (comment)

When building from a worktree, the common git directory was not mounted
in docker.

This lead to the version not being set and issues with git-cache in
ubuntu bionic that could not execute the 'git hash-object' command.

(cherry picked from commit 61a3e5d)
@cladmi cladmi added Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … Area: tools Area: Supplementary tools CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation labels Oct 31, 2018
@cladmi cladmi requested a review from jcarrano October 31, 2018 14:58
@cladmi
Copy link
Contributor Author

cladmi commented Oct 31, 2018

I will re-run the tests with this one.

@cladmi
Copy link
Contributor Author

cladmi commented Oct 31, 2018

Both test indeed failed with the release branch and worked with this PR.

@cladmi cladmi added this to the Release 2018.10 milestone Nov 1, 2018
Copy link
Contributor

@jcarrano jcarrano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works.

@jia200x
Copy link
Member

jia200x commented Nov 2, 2018

let's wait for Murdock. Thanks for the backport and testing!

@cladmi
Copy link
Contributor Author

cladmi commented Nov 2, 2018

Murdock is green. Thank you for the review of the backport too.

@cladmi cladmi merged commit 488771f into RIOT-OS:2018.10-branch Nov 2, 2018
@cladmi cladmi deleted the backport/2018.10/pr/make/docker/handle_worktree branch November 2, 2018 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … Area: tools Area: Supplementary tools CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants