CCM_Matrix app : scipy KDTree multithread worker crash

[SoftwareLiteracy]

The CCM_Matrix.py application wraps CCM() in a concurrent.futures ProcessPoolExecutor context. CCM() itself is run parallelized for forward and reverse mapping in a multiprocessing Pool context.

Internally, the Simplex function in CCM uses scipy KDTree to find and query neighbors. KDTree.query() is multithreaded.

On a hyperthreaded arch this may result in a crash:

Stack trace of thread 3462316:
#0  0x00007fb79804af16 __futex_abstimed_wait_common64 (libc.so.6 + 0x85f16)
#1  0x00007fb79804d8bc __pthread_cond_wait_common (libc.so.6 + 0x888bc)
#2  0x00007fb7983be167 take_gil (/3.12.4/lib/libpython3.12.so.1.0 + 0x1e9167)
#3  0x00007fb7983e1062 PyEval_RestoreThread (/3.12.4/lib/libpython3.12.so.1.0 + 0x20c062)

#4  0x00007fb694d6dd0e _ZL64__pyx_pw_5scipy_7spatial_8_ckdtree_7cKDTree_5query_1_thread_funcP7_objectPKS0_lS0_ (/3.12.4/lib/site-packages/scipy/spatial/_ckdtree.cpython-312-x86_64-linux-gnu.so + 0x3bd0e)

#5  0x00007fb7982e78c6 _PyEval_EvalFrameDefault.cold /3.12.4/lib/libpython3.12.so.1.0 + 0x1128c6)
#6  0x00007fb798429216 _PyObject_VectorcallTstate.lto_priv.4 (/3.12.4/lib/libpython3.12.so.1.0 + 0x254216)
#7  0x00007fb798428d57 method_vectorcall (/3.12.4/lib/libpython3.12.so.1.0 + 0x253d57)
#8  0x00007fb798509ca3 thread_run (/3.12.4/lib/libpython3.12.so.1.0 + 0x334ca3)
#9  0x00007fb7984c9c04 pythread_wrapper (/3.12.4/lib/libpython3.12.so.1.0 + 0x2f4c04)
#10 0x00007fb79804e1f5 start_thread (libc.so.6 + 0x891f5)
#11 0x00007fb7980ce89c __clone3 (libc.so.6 + 0x10989c)

Currently the call to KDTree.query in pyEDM sets workers = -1 to use all available CPU's.

Propose to add a parameter allowing user to set the number of worker threads used in KDTree.query