Attempting to cross-compile linux kernel, getting stuck on inline assembly `.incbin` file not found error

[EvanKrall]

I'm working on a Raspberry Pi project, and I often need to recompile the kernel for it. I've been cross-compiling from my (fairly old; i7-6500U) intel box, so llama seems like it should save me a bunch of time.

I've got this sorta working. I've got a directory _toolchain with two scripts in it:

_toolchain/activate (which you source like you would to activate a python virtualenv):

_OLDPATH="${_OLDPATH:-$PATH}"
_OLDPS1="${_OLDPS1:-$PS1}"
export PATH="$(dirname $(readlink -f "${BASH_SOURCE[0]}")):$HOME/go/bin:$PATH"
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
export LLAMACC_FUNCTION=gcc-aarch64-linux-gnu
export LLAMACC_LOCAL_CC=/usr/bin/aarch64-linux-gnu-gcc
export LLAMACC_LOCAL_CXX=/usr/bin/aarch64-linux-gnu-g++
export KBUILD_BUILD_TIMESTAMP='ccache'  # probably not relevant to llama

PS1="${_OLDPS1}\[\033[0m\](llamacc arm64) "

function deactivate() {
	export PATH="$_OLDPATH"
	PS1="$_OLDPS1"
	unset ARCH
	unset CROSS_COMPILE
	unset KBUILD_BUILD_TIMESTAMP
}
# ... and some other functions helpful for my own purposes

_toolchain/aarch64-linux-gnu-gcc:

#!/bin/bash
exec llamacc "$@"

As well as this Dockerfile at images/gcc-aarch64-linux-gnu-jammy/Dockerfile:

FROM ghcr.io/nelhage/llama as llama
FROM ubuntu:jammy
RUN apt-get update && apt-get -y install ca-certificates gcc-11-aarch64-linux-gnu gcc-aarch64-linux-gnu && apt-get clean

RUN cd /usr/bin/; for exe in aarch64-linux-gnu-*; do ln --force -s "/usr/bin/$exe" "${exe##aarch64-linux-gnu-}"; done
RUN cd /usr/bin/; ln --force -s /usr/bin/aarch64-linux-gnu-gcc cc
COPY --from=llama /llama_runtime /llama_runtime
WORKDIR /
ENTRYPOINT ["/llama_runtime"]

(and, of course, gcc-aarch64-linux-gnu installed locally on my local box, which is also running jammy.)

Much of the build seems to work, but I'm getting stuck building the configs module:

$ (llamacc arm64) make -j100 Image modules dtbs
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CC [M]  kernel/configs.o
/tmp/cczL3rE5.s: Assembler messages:
/tmp/cczL3rE5.s:9: Error: file not found: kernel/config_data.gz
Running llamacc: invoke: exit 1
make[1]: *** [scripts/Makefile.build:288: kernel/configs.o] Error 1
make: *** [Makefile:1868: kernel] Error 2

LLAMACC_REMOTE_ASSEMBLE is unset, so I'm not sure why it's trying to assemble remotely. I guess because kernel/configs.c has an inline assembly block:

/*
 * "IKCFG_ST" and "IKCFG_ED" are used to extract the config data from
 * a binary kernel image or a module. See scripts/extract-ikconfig.
 */
asm (
"	.pushsection .rodata, \"a\"		\n"
"	.ascii \"IKCFG_ST\"			\n"
"	.global kernel_config_data		\n"
"kernel_config_data:				\n"
"	.incbin \"kernel/config_data.gz\"	\n"
"	.global kernel_config_data_end		\n"
"kernel_config_data_end:			\n"
"	.ascii \"IKCFG_ED\"			\n"
"	.popsection				\n"
);

With LLAMACC_VERBOSE=1: gist because github doesn't like comments over 65k

Notably, I don't see anything about kernel/config_data.gz in the LLAMACC_VERBOSE output. The file does exist locally.

Calling LLAMACC_LOCAL=1 make kernel/configs.o succeeds, and seems to allow the rest of the build to continue. 🎉

[nelhage]

Oof. The basic situation here is:

• In order to build a source file in the cloud, llama has to ensure that any other files it depends on are also present in the Lambda instance.
• Llama uses the C preprocessor (via some command-line options exposed by gcc and clang) to discover those dependencies.
• Unfortunately, the assembler also has an include mechanism. This creates a dependency on an external file, which we cannot "see" through the preprocessor.
• That's the primary reason LLAMACC_REMOTE_ASSEMBLE is not enabled by default. That option only affects assembly source files (.S); we still assemble the output of the C compiler remotely, because the C compiler does not use assembly-level include mechanisms.
• But in this case, the kernel developers are writing C code that causes the resulting assembly to have such a mechanism, and so we break.

The obvious avenues to fix are:

• Somehow add support for sending assembly .include / .incbin dependencies along. I'm not sure how hard that is.
• Somehow detect this situation and fall back to a local build.
  • We could either do that before sending the file to the cloud, or we could detect this particular class of error and have the cloud send back a "rerun locally" error
  • Least satisfyingly, but maybe useful, we could have a general mechanism for the user to opt specific paths or specific command lines out of remote building, so that you could do the previous step by hand.

First things first, llama is awesome, thanks @nelhage!

I'm running into the same issue while trying to rebuild a Ubuntu x86 kernel from their Ubuntu-6.5.0-25.25 tag, no cross-compilation involved. It fails with the following:

/tmp/ccplWhxP.s: Assembler messages:
/tmp/ccplWhxP.s:10: Error: file not found: kernel/kheaders_data.tar.xz

And it's the same problem, really:

 $ git grep kheaders_data.tar.xz
kernel/kheaders.c:23:"  .incbin \"kernel/kheaders_data.tar.xz\" \n"

Tempted to grep for .incbin in llamacc and have it do the local build if it finds it...

I've added the workaround you were suggesting, but taking a big hammer approach of always doing the local build if it fails, keyed off by default under a new environment variable LLAMACC_LOCAL_FALLBACK=1. Tested both with/without and it did for my build.