Shared submodules in the vg tree are at different versions in different copies

[deliciouslytyped]

vg downloads the hopscotch_map repository at runtime. There is no immediately visible reason why this is necessary, and is also inconsistent with the approach of vendoring dependencies in submodules in the deps folder. *

This build time downloading is a problem for builds run in the Nix build sandbox, since to ensure determinism, network access is prohibited.

I would ask, as discussed, if it's technically feasible, to modify the build so that for example, hopscotch_map is also a vendored submodule, since that seems to work.

As a sidenote, interestingly enough, hopscotch_map already seems to be a dependency in some of the other submodules(, and some of the versions of submodules are different (this can be seen by looking at the output of git submodule status --recursive, though I'm not sure if this is actually a problem)).

$ git submodule status --recursive | sed -E 's| ([^/ ]+/)+([^/ ]+) | \2\0|' `#duplicate final component of path so we can sort by submodule name and remove dupes` | sort --stable --unique -k 2,2 -k 1,1 `#https://stackoverflow.com/a/16859296 https://unix.stackexchange.com/a/52819 `
 78e27faf75ff7d20f232f11ffcef65cde43c449d args deps/xg/deps/args (6.2.2-11-g78e27fa)
 de4db870058c37b6094bc5ccb03c9ea45708c855 args deps/mmmultimap/deps/args (6.2.2-17-gde4db87)
 d9d66b6d20d74042da481ed5504fa81c0d79c8ae atomic_queue deps/atomic_queue (d9d66b6)
 58f21c22d310d1c2078aad1a71254749d6f588df backward-cpp deps/backward-cpp (v1.0-93-g58f21c2)
 c38fbfa401600cc81ccda66bfc0da3ea56288d03 bash-tap deps/bash-tap (1.0.2)
 d7db4e882a5bc8b141333a81f9fc640ec0784661 bash-tap deps/gfakluge/bash-tap (1.0.2-1-gd7db4e8)
 36e4fe3eaeef762c831c49cdc01f1a3a2c7a97a4 BBHash deps/BBHash (v1.0.0-13-g36e4fe3)
 6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5 clang deps/libbdsg/bdsg/deps/pybind11/tools/clang (6a00cbc)
 7871939afc4c5ad2902d2b449d92b4295fee83f9 cmake-modules deps/mmmultimap/deps/ips4o/extern/cmake-modules (remotes/origin/develop)
 b4ad14299695668c3659c01d7544f494d19e4b59 dozeu deps/dozeu (b4ad142)
 444808a6f25fa4194defb94e687a7f245fda6618 DYNAMIC deps/DYNAMIC (heads/master)
 73a6b10ecb94ee178fa873797aacf81e0bfdc7db DYNAMIC deps/libbdsg/bdsg/deps/DYNAMIC (remotes/origin/add-chrono-1-g73a6b10)
 d08c68fec94b26663257d24dbb8d94f2ed8935cd elfutils deps/elfutils (elfutils-0.176-37-gd08c68fe)
 75f12d25df9416b9d49b84c70dcc58406afce11a fastahack deps/fastahack (heads/master)
 bb332654766c2177d6ec07941fe43facf8483b1d fastahack deps/vcflib/fastahack (v1.0.0)
 243c9330fdb2dc5e0bbb5e441ba3335678ec7136 fermi-lite deps/fermi-lite (v0.1-10-g243c933)
 1a9b779b93d0b244040274794d402106907b71b7 filevercmp deps/vcflib/filevercmp (1a9b779)
 1b1c6deede9c33c5134c920bdb7a44cc5528e9a7 FlameGraph deps/FlameGraph (v1.0-30-g1b1c6de)
 a6ef318fbd347c53189384aef7f670c0e6ce89a3 fsom deps/vcflib/fsom (a6ef318)
 38ca4120fcb338cb91133a65ff6d33c6138b3f2f gbwt deps/gbwtgraph/deps/gbwt (v1.2-43-g38ca412)
 edf48782555d2819f03f3dfd9aec9296cfafa084 gbwt deps/gbwt (v1.2-47-gedf4878)
 e13a56199587815acc3899c437c31909a3b5399c gbwtgraph deps/gbwtgraph (v0.6-31-ge13a561)
 c6b02c69ca58fa8e9c8faa8ffc4382ea59809fcd gcsa2 deps/gcsa2 (v1.3-3-gc6b02c6)
 4ecab57f2f4178e15bd5584cbd1058c6fc54bb2c gfakluge deps/gfakluge (remotes/origin/header-crash-4-g4ecab57)
 a18e71be36653357df9e7fe6a9ac185153ed1cd5 gfakluge deps/xg/deps/gfakluge (remotes/origin/header-crash-1-ga18e71b)
 4b216c79c91702a96f94c2def6f056fbe496dcbf GFA-spec deps/gfakluge/GFA-spec (1.0-68-g4b216c7)
 c2d90bddc6a2a562ee7750c14351e9ca16a6a37a googletest deps/gbwt/deps/sdsl-lite/external/googletest (release-1.8.0-55-gc2d90bdd)
 d225acc90bc3a8c420a9bcd1f033033c1ccd7fe0 googletest deps/vcflib/googletest (release-1.8.0~6)
 e17bad23ab68cfa9b8668b8298187d0b308b6129 gssw deps/gssw (remotes/origin/ssw-compat)
 848374746a50b3ebebe656611d554cb134e9aeef hopscotch-map deps/mmmultimap/deps/hopscotch-map (v2.3.0-2-g8483747)
 b0de25158da691e6680bb5c6c2a12ede3a4b0f70 hopscotch-map deps/libbdsg/bdsg/deps/hopscotch-map (v2.2.1-5-gb0de251)
 30bc9fdca45e144bd975eb2a2563c1cac43c2ec5 htscodecs deps/htslib/htscodecs (v1.0)
 6d2bfb7031c3237507815a72e917f1f20bff1288 htslib deps/sublinear-Li-Stephens/deps/htslib (1.7-6-g6d2bfb7)
 bd133acf51498431a5c7dfd8aa06ce17ec6d3b96 htslib deps/htslib (1.12)
 fd0f89554459b78c07303e2c8a42acacd6851b46 htslib deps/tabixpp/htslib (1.10.2)
 17ddd578a455fe6c10e3ecde1c67121d4412bcec intervaltree deps/vcflib/intervaltree (v0.1~2)
 22069381cc1bf2df07ee1ff47f6b6073fcfb4508 ips4o deps/ips4o (heads/master)
 840c0d80cb9c8b5e805f9d5693104bcda3ea5e31 ips4o deps/xg/deps/ips4o (840c0d8)
 cf269199fb1ed91751dbdba032339992decf220d ips4o deps/mmmultimap/deps/ips4o (cf26919)
 041145c272711b55f91aa42128b108674a12fd91 jemalloc deps/jemalloc (5.2.1-688-g041145c2)
 f8b1da408e7c4db8c5a29edbff2e114b270e07ce libbdsg deps/libbdsg (v0.3-35-gf8b1da4)
 2b6689d8aa30175712a1e697adeac884fc3720dc libdeflate deps/libdeflate (v1.0-4-g2b6689d)
 0f24acd8de208464769c782119dacf158647f7ed libdivsufsort deps/gbwt/deps/sdsl-lite/external/libdivsufsort (2.0.1-13-g0f24acd)
 0c0697052ef44f71691af6adc9865ead332f32c7 libhandlegraph deps/libhandlegraph (remotes/origin/ordered-chop-1-g0c06970)
 40757e853f952b2cf8cd5aa074fedf84dda10e58 libhandlegraph deps/libvgio/deps/libhandlegraph (remotes/origin/rank-nodes-1-g40757e8)
 521ec5a4201b016227ddeea838e23e28203634f7 libhandlegraph deps/gbwtgraph/deps/libhandlegraph (remotes/origin/tighten-semantics-3-g521ec5a)
 a57a51f0be441b1d734e960a8bbfd8a9e82bc5c2 libhandlegraph deps/libbdsg/bdsg/deps/libhandlegraph (remotes/origin/truncate-2-ga57a51f)
 bb4c9c8b5c39a5baa2fc2e36eac0ccaa4aa879b9 libhandlegraph deps/xg/deps/libhandlegraph (remotes/origin/rank-nodes)
 615a06b5de21f59294d7d806e9f505162ae8c2c1 libVCFH deps/libVCFH (615a06b)
 4378b7e23d6be0e99401da8736584fdbb05153cf libvgio deps/libvgio (remotes/origin/fake-threads)
 98c5eacfb2bf60debd2f9c3a3faf814e8821458c lru_cache deps/lru_cache (heads/master)
 3f86a95c0784d73ce6815237ec33ed25f233b643 mio deps/mio (heads/master)
 d8fb6c995d3d9dc5e5844c5f72d1e7d32163fd0a mmmulti deps/xg/deps/mmmulti (v0.1-22-gd8fb6c9)
 d8fb6c995d3d9dc5e5844c5f72d1e7d32163fd0a mmmultimap deps/mmmultimap (v0.1-22-gd8fb6c9)
 68733cd9a438fafd76f20393f2e68d38c67d9ebe multichoose deps/vcflib/multichoose (v1.0.3-4-g68733cd)
 509b28a092f732a068e2908bb9e359a8562cd32f paryfor deps/mmmultimap/deps/paryfor (v0.1)
 52df64cc1e2ad2633cc9192381c5a8c9021e772a pinchesAndCacti deps/pinchesAndCacti (july2012_0-60-g52df64c)
 e20ae5296b278bb6afb287a2e078a33c88f1c528 progress_bar deps/progress_bar (e20ae52)
 a54eab92d265337996b8e4b4149d9176c2d428a6 pybind11 deps/libbdsg/bdsg/deps/pybind11 (v2.5.0-18-ga54eab9)
 17fe084ee96def426beb62bac9c6fb4d8f741c92 raptor deps/raptor (raptor2_2_0_15-199-g17fe084e)
 010e3e1dc4ac1fdd21ab4561479687fd8bd2990f sdsl-lite deps/gbwtgraph/deps/gbwt/deps/sdsl-lite (v2.3-vgteam-8-g010e3e1)
 2039a78b42dc3c981a5087f433e26c0ac116267f sdsl-lite deps/libbdsg/bdsg/deps/sdsl-lite (v2.1.1-153-g2039a78)
 d4105e78ea42f8aeacb18a5ec8983430ef6190fb sdsl-lite deps/sdsl-lite (v2.3-vgteam-13-gd4105e7)
 ffc8ed5337aa0d01cf4f2c60e619a873e0b0ce7d sdsl-lite deps/gbwt/deps/sdsl-lite (v2.3-vgteam-12-gffc8ed5)
 011accb73af23c2703bd6801cd4285728c264d94 sha1 deps/sha1 (1.3)
 10321a4ab244984b642f22a18a33bc216b7054ff simde deps/fermi-lite/simde (v0.5.0-45-g10321a4)
 2931676b5fc30ab3f782759c75f534a36cd2fad4 simde deps/ssw/src/simde (v0.5.0)
 3a4c5bbc84320d846b35944a0b022b67b386413b simde deps/dozeu/simde (v0.5.0-114-g3a4c5bb)
 8cd136a43bae7ab9b82316179b9cef8887726778 simde src/simde (v0.7.0)
 2610e259611ae4cde8f03c72499d28f03f6d38a7 smithwaterman deps/vcflib/smithwaterman (heads/master)
 2b9152d9c5bed71dffb7f7f6c7a3ec48b058ff2d snappy deps/snappy (1.1.3-8-g2b9152d)
 63b0301bce3861f247684c804fd204c452b8faf2 sonLib deps/sonLib (july2012_0-321-g63b0301)
 a61a6ba7adbc4e3a7545843a72c530bf35604dae sparsehash deps/sparsehash (sparsehash-2.0.3-7-ga61a6ba)
 1c5a285e87b2244383efe0398f9992acd61234e9 sparsepp deps/libbdsg/bdsg/deps/sparsepp (1.1-50-g1c5a285)
 7ac6088af04358843bb1816e8be69974d1973a20 sparsepp deps/sparsepp (1.1-16-g7ac6088)
 f53d459841b632ac6ab0e59e1a0ef6d056034631 ssw deps/ssw (heads/master)
 f1990d91b8da91904d37de6937d5c93c60708165 structures deps/structures (f1990d9)
 4efb2f3933ea8c684fe289e8f07f626f0747436e sublinear-Li-Stephens deps/sublinear-Li-Stephens (4efb2f3)
 21086c3528f026dd73c9cfa0c5a86cf713cbac47 tabixpp deps/tabixpp (v1.1.0-2-g21086c3)
 83e6414fe913433dc176b558a75a27a25cc23ea5 vcflib deps/vcflib (v1.0.0-rc0-596-g83e6414)
 4454d5a7ff00c5ec30629deef2697044e6c4c14d wiki doc/wiki (4454d5a)
 6ee7398f0971aaf73801d7ba76a5b4e8696fd4c7 xg deps/xg (v0.1.0-35-g6ee7398)

* I didn't realize DYNAMIC is also a submodule that is a library, I just assumed it was an oddly named directory for deps that were downloaded at build-time (which is wrong), since it was the only case of this. Should I move this issue to the DYNAMIC repo?

[deliciouslytyped]

Edit: I figured this can't be that hard, I should try to make a PR myself.

cc @nicolaprezza @karasikov because I can't figure out who would know the correct way to set up the includes;
What are these lines supposed to do?:

ExternalProject_Get_property(hopscotch_map INSTALL_DIR)
set(hopscotch_map_INCLUDE "${INSTALL_DIR}/src/hopscotch_map/include/")

include_directories(${PROJECT_SOURCE_DIR}/include)
include_directories(${INSTALL_DIR}/src/hopscotch_map/include)

[deliciouslytyped]

Ok, looking at the output of cmake --trace-expand .. in DYNAMIC/bin, while patching to have the following (i.e. using source_dir):

ExternalProject_Add(hopscotch_map
  SOURCE_DIR "${CMAKE_SOURCE_DIR}/deps/hopscotch_map"
  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} # TODO ADD static build flag
  UPDATE_COMMAND ""
  INSTALL_COMMAND "" 
  BUILD_COMMAND ""
  CONFIGURE_COMMAND "")

yields the following value for hopscotch_map_INCLUDE: /myworkdir/DYNAMIC/bin/hopscotch_map-prefix/src/hopscotch_map/include/

Listing that directory after a build yields:

$ ls hopscotch_map-prefix/src/
hopscotch_map-build  hopscotch_map-stamp

So something is wrong, because the source doesn't get copied there.

[deliciouslytyped]

Open PR: xxsds/DYNAMIC#28

[deliciouslytyped]

To clarify an earlier point, the quote in the original posts shows duplicates with differring versions not only for hopscotch, but other submodules.

[adamnovak]

@deliciouslytyped Is there a way to keep these in sync sensibly? Does CMake maybe have a pattern we can use where you can have several repos, each of which can build in isolation, but which can also build in a project together without introducing multiple versions of shared dependencies? Can we pull in something from Nix to manage our build process?

So far we've been ignoring the multiple-versions problem except when it manifests as CI failures, but you're right that it's a source of trouble waiting to happen.