Multi-GPU QR

[ravankhidirov]

Hi team!

I wanted to kindly ask if there are any plans or timelines for supporting multi-GPU QR decomposition in cupynumeric? Any update on that would be great.

Thank you!

[manopapad]

Hello! This is in currently in development (based on cuSolverMp).

[IlgarBaghishov]

Hi @manopapad, thank you for your reply. Is there an approximate timeline for this? I am asking to know if it would make sense to invest time and try to get it working with cuSolverMp ourselves or just wait until you release it in cupynumeric.

[manopapad]

@aschaffer @mfoerste4 do you have a sense how far along you are?

[aschaffer]

We expect to make it sometime next week. That's assuming we can quickly identify and fix the synchronization issues due to cusolvermp using own stream. Debugging that.

[IlgarBaghishov]

That's great! Thank you so much for your reply.

[IlgarBaghishov]

@aschaffer @manopapad Hope you are doing well! Is there any new timeline for this? Or has this been released already? Thank you!

[manopapad]

Sorry for the delay in responding. We had to upgrade to cuSolverMp to 0.7 to get a bugfix for the multi-GPU QR path, which triggered a lot of churn, as we had to change the initialization logic (cuSolverMp 0.7 moved from UCC to NCCL). Unfortunately we are still seeing crashes in CI, so this isn't quite ready to merge. I'm hoping for mid-October at this point.