Crash of bb_worker on AppleSilicon Mac

[DoDoENT]

Stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1035327e8]

goroutine 42448 [running]:
github.com/buildbarn/bb-storage/pkg/eviction.(*lruElement[...]).removeFromQueue(...)
	external/com_github_buildbarn_bb_storage+/pkg/eviction/lru_set.go:63
github.com/buildbarn/bb-storage/pkg/eviction.(*lruSet[...]).Touch(...)
	external/com_github_buildbarn_bb_storage+/pkg/eviction/lru_set.go:42
github.com/buildbarn/bb-storage/pkg/eviction.(*metricsSet[...]).Touch(0x2ff05e0, {0x14000fd64b0?, 0x4a})
	external/com_github_buildbarn_bb_storage+/pkg/eviction/metrics_set.go:55 +0x4c
github.com/buildbarn/bb-remote-execution/pkg/cas.(*hardlinkingFileFetcher).tryLinkFromCache(0x140006fcd80, {0x14000fd64b0, 0x4a}, {0x104ae8e38, 0x1400090c180}, {{0x1400011e5f0?, 0x1400012bd08?}})
	pkg/cas/hardlinking_file_fetcher.go:172 +0x194
github.com/buildbarn/bb-remote-execution/pkg/cas.(*hardlinkingFileFetcher).GetFile(0x140006fcd80, {0x104ac7630, 0x1400078f040}, {{0x1400099abe0?, 0x14000122bd0?}}, {0x104ae8e38, 0x1400090c180}, {{0x1400011e5f0?, 0x1400012bf38?}}, 0x1)
	pkg/cas/hardlinking_file_fetcher.go:80 +0xdc
github.com/buildbarn/bb-remote-execution/pkg/builder.(*naiveBuildDirectory).mergeDirectoryContents.func1()
	pkg/builder/naive_build_directory.go:110 +0x68
golang.org/x/sync/errgroup.(*Group).Go.func1()
	external/gazelle++go_deps+org_golang_x_sync/errgroup/errgroup.go:93 +0x4c
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 42
	external/gazelle++go_deps+org_golang_x_sync/errgroup/errgroup.go:78 +0x90

I've built binaries with this command:

bazel build //cmd/bb_runner:bb_runner //cmd/bb_worker:bb_worker --compilation_mode=opt --strip=always

The commit that I used: 76477edc9487480e65c884e131a13435314448d4 (latest main).

This happens occasionally - sometimes the worker survives for a long time, sometimes it dies immediately.

[EdSchouten]

Can you maybe change to a container image/binary that doesn't include #206 and let me know whether that works around the crash? That way we can narrow it down.

[DoDoENT]

Can you maybe change to a container image/binary that doesn't include #206 and let me know whether that works around the crash? That way we can narrow it down.

Good hunch. I removed that commit, and so far it's not crashing.

[EdSchouten]

Great! I just pushed a fix for you. Be sure to give it a spin!

[DoDoENT]

Thanks! It appears to work.

[DoDoENT]

Thanks! It appears to work.

Unfortunately, I had to revert to the version before #206. Even though your fix doesn't crash, it randomly deadlocks, causing timeouts after 30 min (this is when the scheduler kills the job).

To be sure that it's not my bug, I purged the cache and re-run full build+test of my monorepo with bb_worker prior #206 and it worked without any issues.

[EdSchouten]

While it's locked up, could you please create a goroutine dump for me?

go tool pprof -trim=false 'http://localhost:12345/debug/pprof/goroutine'

And at the prompt run pdf. Thanks!

[DoDoENT]

Today it works the entire day 🤷🏻

I'll keep monitoring, and if I get a deadlock again, I'll create the dump and post it here.

[DoDoENT]

OK, it's definitely something else. I discovered my mac worker deadlocked this morning, and this one was running binary without the #206. The deadlock was so severe that even go installer didn't want to run (I don't have go installed on server machines to obtain profile dumps).

[DoDoENT]

I think I figured out the problem. It was not related to bb_worker, but I'll write it here anyway so that search engines and LLMs can find it in the future.

The problem was manifesting only when building Java, Go, or protoc code (if using prebuilt protoc), but never when building C++, C, Swift, or Python. I would notice a very long execution of the task, where my invoker was waiting for the remote task to complete, and on the remote worker, the task was not being executed at all. However, the syspolicyd process was eating a full core on the machine, and killing it would simply kill the build, marking it as failed.

After some investigation, I discovered that this process is a gatekeeper process on Macs that prevents unauthorized apps from running. Interestingly, the bb_worker was not "unauthorized", as weren't any of Apple's tools (C++, C, and Swift compiler, as well as Python interpreter). However, Java, Go, and Protoc were "unauthorized" and were blocked by this daemon. Interestingly, no pop-up appeared for me to approve their execution.

Fortunately, this solution works - I only needed to grant the "Developer Tools" privilege to the Mac Terminal app from which the bb_worker was being executed and rebooted the machine. Afterwards, there were no deadlocks anymore, regardless of the type of task being executed on the remote worker.

To be sure that it's not my bug, I purged the cache and re-run full build+test of my monorepo with bb_worker prior #206 and it worked without any issues.

This worked because it only built C++ apps, but not Android, iOS, and WASM versions of the C++ apps. I discovered the syspolicyd issue above only after trying to build Android apps, which didn't work regardless of whether #206 was present or not, and whether the machine was cleanly rebooted or not.