Parallel iterators and notebook mode results in a deadlock

[istoleyurballs]

Hi,
I think I found a bug when using the notebook and rayon features at the same time.

This snippet seems to cause a deadlock.

#[pyfunction]
fn my_function(something: &[u8]) {
  kdam::set_notebook(true);
  kdam::par_tqdm!(something.par_iter()).something();
}

When calling this function inside a jupyter notebook, it just hang indefinitely.

Workarounds

Turning off notebook mode works but defeats the purpose.

Wrapping the iteration inside a py.allow_threads(|| ...) seems to also resolve the issue.

However in this sort of notebook workflow you often iterate over things that come from python, so not holding the GIL during the entire function is often wrong and could result in unexpected memory races.

[istoleyurballs]

Hi, an update on this: the allow_threads workaround doesn't solve it, it just makes it rarer / take longer to appear.

Sometimes it hangs indefinitely in the middle of a task, sometimes it crashes with a "too many files open".
And indeed, the more progress bar are created, the more the amount of file descriptors grows, on my machine it hangs at exactly 1080 file descriptors open in /proc/pid/fd each time (which is approximatively my ulimit -n of 1024) (/proc/pid/fd is filled with anon_inode:[eventfd]),

The problem disappears when using set_notebook(false) (including the growing amount of fd), indicating that this is related to kdam.

Note that the fd amount continue to grow even when re-executing the same cell, making the previous widget disappear, meaning its not just many visible progress bar widgets that's causing it.

I guess those file descriptors are the result of communicating with jupyter, and are improperly freed.

My guess is the Bar instance is leaked in the rayon wrapper and it ends up undetected most of the time, but having the notebook feature makes its noticeable.