Fragile bootstrap (was cp: cannot stat 'obj/mbld/mbld': No such file or directory)

[Earnestly]

• a819f7c

Building after a clean clone with the following:

% ./configure --prefix=/usr
% make -j4 bootstrap            # This happens without -j4 as well

Yields a new error:

[...]
+ ar -rcs obj/lib/regex/libregex.a obj/lib/regex/interp.o obj/lib/regex/ranges.o obj/lib/regex/types.o obj/lib/regex/compile.o
+ /home/earnest/build/staging/myrddin-git/src/mc/muse/muse -o obj/lib/regex/libregex.use -p regex obj/lib/regex/interp.use obj/lib/regex/ranges.use obj/lib/regex/types.use obj/lib/regex/compile.use
+ /home/earnest/build/staging/myrddin-git/src/mc/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/subtest.myr
+ /home/earnest/build/staging/myrddin-git/src/mc/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/test.myr
+ /home/earnest/build/staging/myrddin-git/src/mc/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/deps.myr
+ /home/earnest/build/staging/myrddin-git/src/mc/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/main.myr
+ ld --gc-sections -o obj/mbld/mbld /home/earnest/build/staging/myrddin-git/src/mc/rt/_myrrt.o obj/mbld/deps.o obj/mbld/main.o obj/mbld/util.o obj/mbld/cpufeatures.o obj/mbld/libs.o obj/mbld/syssel.o obj/mbld/config.o obj/mbld/opts.o obj/mbld/subtest.o obj/mbld/types.o obj/mbld/test.o obj/mbld/install.o obj/mbld/parse.o obj/mbld/build.o -Lobj/lib/thread -lthread -Lobj/lib/bio -lbio -Lobj/lib/regex -lregex -Lobj/lib/std -lstd -Lobj/lib/sys -lsys
ld: cannot find -lthread
ld: cannot find -lstd
ld: cannot find -lsys
+ true
cp obj/mbld/mbld xmbld
cp: cannot stat 'obj/mbld/mbld': No such file or directory
make: *** [Makefile;26: bootstrap] Error 1

Interestingly, if I run make bootstrap again, it completes without issue.

[oridb]

Hm. What platform? (The change you cited only affects plan 9).

It's likely that the bootstrap just wasn't regenerated correctly. If you install, and then run make genbootstrap, do you get differences? If you do, can you send a pull request?

[Earnestly]

Ah, I thought you may remember me. This is on Linux, the linked commit is the current commit I was using, and yes running make genbootstrap seems to fix the issues.

This is the diff I get afterwards:

diff --git a/mk/bootstrap/bootstrap+Linux-x86_64.sh b/mk/bootstrap/bootstrap+Linux-x86_64.sh
index 8bc30cc3..fcaa108c 100755
--- a/mk/bootstrap/bootstrap+Linux-x86_64.sh
+++ b/mk/bootstrap/bootstrap+Linux-x86_64.sh
@@ -1,6 +1,24 @@
 #!/bin/sh
 # This script is generated by genbootstrap.sh
 # to regenerate, run "make bootstrap"
+mkdir -p obj/lib/bio
+mkdir -p obj/lib/bld.sub
+mkdir -p obj/lib/crypto
+mkdir -p obj/lib/date
+mkdir -p obj/lib/escfmt
+mkdir -p obj/lib/fileutil
+mkdir -p obj/lib/flate
+mkdir -p obj/lib/http
+mkdir -p obj/lib/inifile
+mkdir -p obj/lib/iter
+mkdir -p obj/lib/json
+mkdir -p obj/lib/math
+mkdir -p obj/lib/regex
+mkdir -p obj/lib/std
+mkdir -p obj/lib/sys
+mkdir -p obj/lib/testr
+mkdir -p obj/lib/thread
+mkdir -p obj/mbld
 pwd=`pwd`
 set -x
        $pwd/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/config.myr

[refi64]

Ah, I thought you may remember me.

Something about this is just so sad...

[Earnestly]

I've decided to just include make genbootstrap in the usual build as it avoids this issue and any future issues that might originate from improperly generated bootstraps.

[oridb]

Keep in mind that genbootstrap assumes you have a working compiler already available. Bootstrap is there only to get you to a working compiler and build system.

At some point, I'm thinking of just having the bootstrap pull down prebuilt/presigned tarballs.

[Earnestly]

Oh, that's no good then. Ideally the bootstrap should not download anything. It should be possible to download everything necessary for a build and then not require network access beyond that point.

edit: some packaging systems will enforce this by using a separate network namespace.

[oridb]

The root of the problem is that the libraries are built with mbld; Mbld depends on the libraries. The bootstrap script hard codes a line by line replay of what the mbld that generated the bootstrap script did.

We need some way of getting a working mbld to build things. That mbld can come from anywhere, but it needs to come from somewhere. Options include a -bootstrap tarball, be generated via the mk/bootstrap/* script, be committed into the repo for each architecture, or be baked into a release tarball that's used to build from git.

I'm not sure what's the least painful. The bootstrap script keeps getting broken when the libraries change, though, and I'm becoming less of a fan of keeping it up to date.

[oridb]

(Also, I definitely remember you, but I don't memorize what platform everyone uses -- and even if I did, people sometimes molt, and come back with a new computer.)

[Earnestly]

Heh, fair enough. Not sure what to suggests here but keeping tarballs in the git repo sounds like a bad call. The chezscheme guys did this and eventually ended up with a repo over 1G in size.

[oridb]

What I'm starting to lean towards is that when I do make release, it should pack a prebuilt copy of mbld for each supported architecture/os combo; that'll use a few megs in the release, but internet is fast these days. Then, I'll require a previously installed version to build git. I can put those up online and have a convenience script for bootstrapping that downloads off the internet.

That's a bit inconvenient for someone to build git and demand no internet accesss, but it's probably workable, and a lot easier to maintain. Including bootstrap scripts with the releases would probably be a solution.

[oridb]

...huh. that's another thought: Adding a commit id to the bootstrap script; when I bootstrap from git, I check out the commit and build that code, which should be in sync with the script. Once I do that, I'd have a working mbld, and can check out the previous branch and rebuild using the bootstrapped mbld.

[refi64]

What about just doing a pre-commit hook that makes sure the bootstrap is up to date?