Emitting a `log` message during interpreter finalization will cause a panic

[ritikmishra]

In my project, I have a Rust struct that will print a log message when it is dropped. In my Python code, this Rust struct lives in global scope and is not deallocated by the interpreter until the Python process exits.

When the Python process exits and drops my Rust struct, pyo3_log::Logger panics when trying to log the message. This is because it tries to acquire the GIL, but since the interpreter is in the process of finalization, the GIL is not available, so pyo3::Python::with_gil panics

Due to PyO3/pyo3#2102, this panic manifests as a SIGABRT being sent to the thread.

--

I think one potential solution is to check that the interpreter is in an initialized state before trying to acquire the GIL using pyo3::ffi::Py_IsInitialized -- however the function is unsafe and I'm not familiar enough with Python internals to know when it is safe to call.

[vorner]

Hello

That Py_IsInitialized doesn't seem like a proper solution. The state could change between checking it and then calling the with_gil. I think we want to have a solution, but this one doesn't seem to be the correct one.

But I wonder what the expected behavior would even be, in such case? Just drop the message? For that we would probably need some kind of with_gil_fallible that can refuse to run instead of panicking. I don't see one at a first glance, would you mind asking the pyo3 folks about it or if there's other, preferred, method?

[arielb1]

Python 3.14 supposedly comes with a change that makes this hang (python/cpython#87135).

What we could do here is spawn a Python thread to handle logging, and marshal all logs to it using a Rust queue. This way, the GIL will be locked when going from the Rust queue to the Python thread, which will lead to the pthread_exit being called with no bad Rust code on the stack.

in Rust 1.84, this causes a crash in more cases (AFAICT, unless you have debuginfo = 0, even in panic = abort mode).

[vorner]

What we could do here is spawn a Python thread to handle logging, and marshal all logs to it using a Rust queue.

This seems rather wrong. For one, it really changes the reliability of logging ‒ that is, if you application crashes hard, you can be missing the logs just leading to that crash, because they are still sitting in that queue. While delayed/asynchronous logging has its place, a library probably shouldn't just do that behind the user's back. Furthermore, just spawning some threads behind the scenes (ones that don't shut down) seems rather rude thing to me; sometimes I want to write a single-threaded application and there may be good reasons not to have more than one thread.

[NilsIrl]

With the thread hanging there is now no way to detect that this took place from another thread. (whereas previously it was possible to pthread_join on a that called pthread_exit)

It doesn't seem like there's any way to recover when this happens...