Thread creation fails under systemd's `MemoryDenyWriteExecute=true`

[bluetech]

systemd has a hardening flag for services MemoryDenyWriteExecute, described in the manpage as folllows:

Takes a boolean argument. If set, attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable, are prohibited. Specifically, a system call filter is added (or preferably, an equivalent kernel check is enabled with prctl(2)) that rejects mmap(2) system calls with both PROT_EXEC and PROT_WRITE set, mprotect(2) or pkey_mprotect(2) system calls with PROT_EXEC set and shmat(2) system calls with SHM_EXEC set. Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers. This option improves service security, as it makes harder for software exploits to change running code dynamically. [...]

When using this option with uv's python (3.14, Linux, x86-64, gnu), thread creation from Python fails:

$ systemd-run -p MemoryDenyWriteExecute=true -t ~/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/bin/python -c 'import threading; threading.Thread().start()'
Running as unit: run-p344916-i344917.service
Press ^] three times within 1s to disconnect TTY.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import threading; threading.Thread().start()
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/ran/.local/share/uv/python/cpython-3.14.0-linux-x86_64-gnu/lib/python3.14/threading.py", line 1004, in start
    _start_joinable_thread(self._bootstrap, handle=self._os_thread_handle,
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           daemon=self.daemon)
                           ^^^^^^^^^^^^^^^^^^^
RuntimeError: can't start new thread

This doesn't happen with system Python (tested on Arch Linux with Python 3.14 and Debian trixie with Python 3.13):

$ systemd-run -p MemoryDenyWriteExecute=true -t python -c 'import threading; threading.Thread().start()'
Running as unit: run-p345006-i345007.service
Press ^] three times within 1s to disconnect TTY.

My initial thought was that this is due to JIT but sys._jit.is_enabled() returns false, and also has nothing to do with threads.

I will try to investigate why this happens when I have some time, but maybe someone has an idea.

[jjhelmus]

This may be related to the lack of hardening flags during the build, see #837. In particular the lack of -z noexecstack and -z relro can result in writable executable memory segments.

[bluetech]

Right, executable stack makes sense, thanks. I ran a pmap -p <pid> on a Python which runs a sleeping thread, and I see this mapping for the new thread's stack:

00007f04a2504000   8196K rwx--   [ anon ]

Reading here https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart, I don't think -z noexecstack should be strictly necessary though, basically it's the default unless there is something preventing it (e.g. same assembly object which doesn't have the gnu note). Not sure how to check what this is. AFAICS, Arch doesn't set -z noexecstack in its default flags (makepkg.conf), but the Arch python binary does have GNU_STACK = RW ELF header; the python-build-standalone doesn't.

Maybe python-build-standalone can set -z noexecstack? I don't think it should affect performance, and since other Python distributions use it, compatibility should be OK hopefully.

[geofft]

Haven't investigated, but I'd expect there are a number of assembly files that transitively end up in the object from statically linking libraries (I'd specifically suspect OpenSSL). A manual -Wl,-z,noexecstack is probably reasonable for us to do.

It would also be good to add a validation test for this.

[bluetech]

I looked into it more, here is what I found so far:

• Only Python >= 3.12 has the problem, Python 3.11 doesn't - confirmed from uv-managed python3.11 binary
• Debug builds don't have the issue - confirmed both downloading from github and building locally
• LTO builds don't have the issue - there is no x86-64 version of this on github, confirmed from building locally
• LTO+PGO builds have the issue - confirmed in version from github
• PGO builds - there is no version of this on github, unfortunately I couldn't build this locally because it segfaults for me during this step. (Not an OOM, just a segfault in the build python).

I checked all of the .a archives and (if I checked correctly) all files in all archives have the .note.GNU-stack note. I therefore don't think it's due to openssl or such.

I checked the .o files in debug build and all files have GNU_STACK (good).

I tried to check the .o files in PGO build, however in this build the files are llvm bitcode which I'm not sure what to do with. However the issue is not due to llvm bitcode itself because LTO-only build is OK.

I'm therefore pretty sure the issue is isolated the PGO builds. And since it only happens starting from Python 3.12, the main suspect is BOLT which according to the doc is enabled by the pgo option since 3.12.

Ran out of time for now but hopefully soon I can confirm this. Due to the segfault it's hard to test with python-build-standalone but maybe it's an upstream cpython issue with --enable-bolt so I'll try to reproduce there.

I did search for "llvm bolt executable stack" and found this issue llvm/llvm-project#174191 which might be related (I am on glibc 2.42), though TBH I don't fully understand what is written there.

[jjhelmus]

@bluetech Thanks for the investigation. I was able to replicate the LTO+PGO build having the issue and a segfault in the PGO only builds.

Some things I tried:

• Adding -Wl,-z,noexecstack (alone or in addition to -Wl,-z,relro and -Wl,-z,now) to the flags used to build CPython and dependencies - still exhibit the issue.
• Disabling BOLT in build-cpython.sh (remove --enable-bolt) - does not have the issue, confirms that BOLT is a root cause.

[bluetech]

Thanks for confirming @jjhelmus.

I tried to test with cpython directly using ./configure --enable-optimizations --enable-bolt using cpython git and llvm-bolt from git, unfortunately the binary segfaults.

I also tried patching out the -use-gnu-stack BOLT flag (added in cpython) and build 3.14 pgo+lto, but it still has the issue.

At this point, without having more knowledge about BOLT, the only thing I can think to check is to build a simple C program which creates a pthread, try to optimize it with BOLT, and see if the thread has an executable stack. I don't think I'll have time to try it soon though.