-
Notifications
You must be signed in to change notification settings - Fork 9
Description
So I did some testing and it seems as though adding support for ray should be relatively straight forward.
I've only tested this locally as I don't have a remote machine to set up on right now. I really don't know how the data list will be appended to in a remote context. Something definitely worth investigating I think. To which, if you have an extra ray server laying around and don't mind sharing it I'd be more than happy to test it out.
The following will enable joblib to work with ray locally.
From the shell
ray start --head --port=6379At the top of demo.py
import ray
ray.init(address='auto', _redis_password='5241590000000000')
...
...
runner = Runner(backend="ray", n_jobs=-1)And at the beginning of Runner._run
if self.backend == "ray":
try:
from ray.util.joblib import register_ray
register_ray()
except ImportError as e:
import sys
raise type(e)(
str(e) + "\nRay backend must be installed"
).with_traceback(sys.exc_info()[2])This should leave the memo package decoupled from ray so it's optional for the user. We could also catch the import error and default back to joblibs loky but my feeling is that if someone passes the backend as ray then failing would be better then raising a warning and continuing running locally. Your thoughts?