Skip to content

Add support for optional ray backend #26

@bradday4

Description

@bradday4

So I did some testing and it seems as though adding support for ray should be relatively straight forward.
I've only tested this locally as I don't have a remote machine to set up on right now. I really don't know how the data list will be appended to in a remote context. Something definitely worth investigating I think. To which, if you have an extra ray server laying around and don't mind sharing it I'd be more than happy to test it out.

The following will enable joblib to work with ray locally.

From the shell

ray start --head --port=6379

At the top of demo.py

import ray
ray.init(address='auto', _redis_password='5241590000000000')
...
...
runner = Runner(backend="ray", n_jobs=-1)

And at the beginning of Runner._run

  if self.backend == "ray":
      try:
          from ray.util.joblib import register_ray
          register_ray()
      except ImportError as e:
          import sys
          raise type(e)(
              str(e) + "\nRay backend must be installed"
          ).with_traceback(sys.exc_info()[2])

This should leave the memo package decoupled from ray so it's optional for the user. We could also catch the import error and default back to joblibs loky but my feeling is that if someone passes the backend as ray then failing would be better then raising a warning and continuing running locally. Your thoughts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions