[multiprocessing] Handle died subprocesses

[ddelange]

Hi 👋

Please consider the following example, where a subprocess tries to allocate more RAM than is available on the system. This snippet will hang/never finish/not raise an exception.

from pathos.multiprocessing import ProcessPool
from random import random

def outofmemory(*args):
    mem = []
    while True:
        mem.append([random() for _ in range(10000)])

for nope in ProcessPool(2).imap(outofmemory, [1, 2]):
    pass

In kubernetes clusters (also local dockerd if you set a container memory limit), subprocesses in a Pod will be SIGKILL'ed at any time and without warning to prevent PID 1 of the Pod going Out of Memory based on its resources.limits.memory spec (ref).

Can iterating over the imap raise an exception when one of the subprocesses dies unexpectedly (so not due to maxtasksperchild)?

Originally posted by @thoughtfuldata in ddelange/mapply#43