[Question] Multi trials in one woker on GPU using Ray tune

[PierrePeng]

Question

Hi all @glvov-bdai ,

I'm using ray tune with this tutorial.

It seems that the paramter about num_woker_per_node means you have how much gpu in your computer or your node.

If that right, only one trial can run in a gpu? It's so weird. Because gpu can't run fully in some cases but it can only run a trial. And why have this restriction?

Many thanks!

[PierrePeng]

@garylvov

[RandomOakForest]

Thank you for posting this. We will review what's the current state of this.

[garylvov]

Hi @PierrePeng sorry for my delayed response. We have a PR in progress that may clear up some of your confusion with provisioning: #2847 (specifying from yaml file)

I believe on main, you are right that only one trial can run per GPU (at one time, you can queue multiple jobs for one GPU). I initially tried to set it up so you can have concurrent runs on one GPU (GPU resource allocation would be specified as a float instead of an int), but I found that the Isaac sim instances didn't really parallelize across one GPU at the time. I can revisit this as I suspect it may actually work with some extra effort. However, at the time, when I tried to do two Isaac sim in parallel to use the entire GPU, the training on the GPU for the parallel instances would somehow interfere, and they would run in series instead. I suspect with the new PR it might work

Id be happy to review and test any PRs that add this functionality. Im trying the new PR I linked above today, I'll try to do a few tests to confirm.

Sorry for any typos sent from my phone

[PierrePeng]

Thanks for your efforts and looking forward the new PR!

Beside that, I'm wondering why you dont use early stop strategy on the trials?

[garylvov]

I think at the time I just didn't set it up while focusing on the repeater and the Optuna Search. I think it should be easy enough to add though feel free to add a PR!

[PierrePeng]

Many thanks @garylvov ! I will try to set it up as soon as possible.

[garylvov]

@PierrePeng RE:
#3270