Pickle error with ctypes shared library

[sjwhitak]

C code:

#include <stdio.h>
int c_add(int x, int y) {
	printf("%d+%d=%d\n", x, y, x+y);
	return x + y;
}

The C code is compiled with:

gcc -shared -fPIC -o test.so test.c

Python code:

import ctypes
from pathos.multiprocessing import ProcessingPool as Pool

lib = ctypes.CDLL('./test.so')
c_add = lib.c_add
c_add.argtypes = [ctypes.c_int, ctypes.c_int]
c_add(2, 2) # Verify the shared library works

# Initialize pool after loading ctypes library
pool = Pool(2)

# Calling C function in parallel
c_res_prl = pool.map(c_add, [i for i in range(10)], [i for i in range(10)]) # fails

The output is:

$ python test.py
C: 2+2=4
Traceback (most recent call last):
  File "/home/sjwhitak/test.py", line 16, in <module>
    c_res_prl = pool.map(c_add, x, y) # fails
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/pathos/multiprocessing.py", line 154, in map
    return _pool.map(star(f), zip(*args), **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/multiprocess/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/multiprocess/pool.py", line 540, in _handle_tasks
    put(task)
    ~~~^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 428, in dump
    StockPickler.dump(self, obj)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 484, in dump
    self.save(obj)
    ~~~~~~~~~^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 920, in save_tuple
    save(element)
    ~~~~^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 905, in save_tuple
    save(element)
    ~~~~^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 905, in save_tuple
    save(element)
    ~~~~^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 2013, in save_function
    _save_with_postproc(pickler, (_create_function, (
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            obj.__code__, globs, obj.__name__, obj.__defaults__,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            closure
            ^^^^^^^
    ), state), obj=obj, postproc_list=postproc_list)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 1133, in _save_with_postproc
    pickler.save_reduce(*reduction)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 690, in save_reduce
    save(args)
    ~~~~^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 905, in save_tuple
    save(element)
    ~~~~^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjwhitak/.conda/envs/test/lib/python3.13/pickle.py", line 576, in save
    rv = reduce(self.proto)
ValueError: ctypes objects containing pointers cannot be pickled

My environment was a fresh setup:

conda create -n test python=3.13.2
pip install pathos

This creates the following from pip list:

Package      Version
------------ -------
dill         0.3.9
multiprocess 0.70.17
pathos       0.3.3
pip          25.0
pox          0.3.5
ppft         1.7.6.9
setuptools   75.8.0
wheel        0.45.1

[sjwhitak]

As an update,

import ctypes
from multiprocess import Process

lib = ctypes.CDLL('./test.so')
c_add = lib.add
c_add.argtypes = [ctypes.c_int, ctypes.c_int]
c_add(2, 2)

x = [i for i in range(10)]
y = [i for i in range(10)]

pList = []
for i in range(10):
    p = Process(target=c_add, args=(x[i], y[i]))
    p.start()
p.join()
print('Done!')

This functions. Something internal to the Pool call seems to be goofy maybe?

[mmckerns]

Thanks for reporting. I'll look into it. Just FYI, the first thing to do generally for pickling issue is to look at the pickle trace -- run both code snippets with dill.detect.trace(True). Also just to note: pathos.multiprocessing.ProcessingPool is identical to pathos.pools. _ProcessPool -- both being deprecated, as opposed to pathos.pools.ProcessPool. Also, pathos.helpers.ProcessPool should be identical to multiprocess.pool.Pool. Also, you can try serialization variants like dill.settings['recurse'] = True.

REF: https://stackoverflow.com/questions/79469633

[sjwhitak]

Yes, I made that stackoverflow question (didn't see the dupe, typical...). If you're wondering, the code was fixed with this very small tweak:

import ctypes
from pathos.pools import ProcessPool as Pool

lib = ctypes.CDLL('./test.so')
c_add = lib.add
c_add.argtypes = [ctypes.c_int, ctypes.c_int]

def wrapper(x, y):
    return c_add(x, y)

pool = Pool(2)
x = [i for i in range(10)]
y = [i for i in range(10)]

# Calling C function in parallel
c_res_prl = pool.map(wrapper, x, y)

I figure you ought to know this as well, maybe it can be worked into pathos/multiprocess! 😄 The SO question was to find an immediate workaround.

I do see a lot of extra print statements with dill trace turned on, so I'll try to see what's going on with my real case. SO's simple fix doesn't work on what I'm actually working on so I need to familiarize myself more with dill it seems. Currently I'll be trying to make a minimum viable solution to get that figured out. Not rushing at all, just thought you'd want to be aware and I'll try to be helpful if possible. Thanks!