[LLVM] `-DLLVM_VERSION_SUFFIX` behaves differently on macOS than on other Unix-like systems

[vadorovsky]

Building LLVM with -DLLVM_VERSION_SUFFIX=-bpf-linker-0 and -DLLVM_BUILD_LLVM_DYLIB=ON behaves differently on macOS than on other Unix-like systems.

With:

"cmake" "-S" "/home/vad/src/llvm-project/llvm" "-B" "./llvm-build" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=RelWithDebInfo" "-DLLVM_BUILD_LLVM_DYLIB=ON" "-DLLVM_ENABLE_ASSERTIONS=ON" "-DLLVM_ENABLE_PROJECTS=" "-DLLVM_ENABLE_RUNTIMES=" "-DLLVM_INSTALL_UTILS=ON" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DLLVM_TARGETS_TO_BUILD=BPF" "-DLLVM_USE_LINKER=lld" "-DLLVM_VERSION_SUFFIX=-bpf-linker-0" "-DCMAKE_C_COMPILER=clang" "-DCMAKE_CXX_COMPILER=clang++" "-DCMAKE_INSTALL_PREFIX=./llvm-install" "-DCMAKE_INSTALL_LIBDIR=lib"

macOS produces unversioned library file libLLVM.dylib with a symlink with suffix libLLVM-22-bpf-linker-2.dylib:

lrwxr-xr-x    1 vad  staff         13 Mar 26 18:13 libLLVM-22-bpf-linker-0.dylib -> libLLVM.dylib
-rwxr-xr-x    1 vad  staff   70060144 Mar 26 18:12 libLLVM.dylib

Linux produces library file with suffix libLLVM.so.22.1-bpf-linker-0 with two symlinks (one with suffix ending with .so, one unversioned):

lrwxrwxrwx 1 vad vad        28 Mar 26 18:52 libLLVM-22-bpf-linker-0.so -> libLLVM.so.22.1-bpf-linker-0
lrwxrwxrwx 1 vad vad        28 Mar 26 18:52 libLLVM.so -> libLLVM.so.22.1-bpf-linker-0
-rwxr-xr-x 1 vad vad 793501688 Mar 26 18:51 libLLVM.so.22.1-bpf-linker-0

I'm working on https://github.com/aya-rs/bpf-linker where we build LLVM from source in CI. To make the LLVM build fast and possible on free GitHub runners, we enable only the BPF target. We produce both static and shared libraries, the latter being useful for compiling bpf-linker binaries with different feature set without each of them having libLLVM embedded (to save some disk space on CI runners). When running tests, we pass our LLVM install directory via LD_LIBRARY_PATH/DYLD_LIBRARY_PATH.

On Linux, in order to avoid clang and rustc from picking our LLVM, we remove the libLLVM.so symlink and ensure that only bpf-linker uses the library with our custom suffix.

Unfortunately, such trick is not possible on macOS. We could in theory rename libLLVM.dylib to something we want, but then we would likely need to also modify the rpath.

What would work great for us is if macOS was doing exactly the same thing as other Unix-like systems. The "fix" would be to remove NOT APPLE from:

llvm-project/llvm/cmake/modules/AddLLVM.cmake

Lines 799 to 806 in 0027f6f

	# Set SOVERSION on shared libraries that lack explicit SONAME
	# specifier, on *nix systems that are not Darwin.
	if(UNIX AND NOT APPLE AND NOT ARG_SONAME)
	set_target_properties(${name}
	PROPERTIES
	# Since 18.1.0, the ABI version is indicated by the major and minor version.
	SOVERSION ${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}${LLVM_VERSION_SUFFIX}
	VERSION ${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}${LLVM_VERSION_SUFFIX})

I think it would also help the Rust team. Currently, the Linux Rust toolchains contain only a versioned .so file:

$ ls rust-dev-nightly-x86_64-unknown-linux-gnu/rust-dev/lib
╭───┬────────────────────────────────────────────────────────────────────────────────────────────┬─────────┬──────────┬──────────────╮
│ # │                                            name                                            │  type   │   size   │   modified   │
├───┼────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼──────────┼──────────────┤
│ 0 │ rust-dev-nightly-x86_64-unknown-linux-gnu/rust-dev/lib/libLLVM-22-rust-1.96.0-nightly.so   │ symlink │     35 B │ 16 hours ago │
│ 1 │ rust-dev-nightly-x86_64-unknown-linux-gnu/rust-dev/lib/libLLVM.so.22.1-rust-1.96.0-nightly │ file    │ 199.5 MB │ 16 hours ago │
╰───┴────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴──────────┴──────────────╯

But the macOS nightly toolchains (which recently switched to linking libLLVM dynamically on macOS, see rust-lang/rust#151063) the .dylib file is unversioned:

$ ls rust-dev-nightly-aarch64-apple-darwin/rust-dev/lib/
╭───┬──────────────────────────────────────────────────────────────────┬──────┬──────────┬──────────────╮
│ # │                               name                               │ type │   size   │   modified   │
├───┼──────────────────────────────────────────────────────────────────┼──────┼──────────┼──────────────┤
│ 0 │ rust-dev-nightly-aarch64-apple-darwin/rust-dev/lib/libLLVM.dylib │ file │ 139.5 MB │ 16 hours ago │
╰───┴──────────────────────────────────────────────────────────────────┴──────┴──────────┴──────────────╯

Therefore it's pretty easy to break cargo +nightly by adding a directory with wrong LLVM version in DYLD_LIBRARY_PATH'.