Add a `rapids doctor` check to verify CUDA toolkit libraries are findable and are version-consistent

[jayavenkatesh19]

RAPIDS libraries can often install successfully via pip or conda but fail at runtime due to the underlying CUDA toolkit not being setup properly. Some scenarios which I have personally come across of this:

• shared libraries (like libcudart.so or libnvrtc.so) are not findable at runtime. Pip wheels previously did not provide these (although with cupy 14, things looks much better). And any setup with a preinstalled CUDA toolkit could have an incorrect configuration.
• CUDA toolkit version (either from pip/conda installations or CUDA_HOME and /usr/local/cuda symlink resolution) does not match with GPU driver's CUDA version.
• The scenario above is further exacerbated by libraries hardcoding /usr/local/cuda as a fallback search path, so a stale symlink loads wrong libraries.

A check on rapids-cli can be added which checks for

• discoverability of shared libraries via cuda-pathfinder
• version consistency between these libraries found the cuda-pathfinder, the GPU driver, the /usr/local/cuda symlink and the CUDA_HOME/CUDA_PATH environment variables (if present). Mismatch on major versions is an automatic error but I am curious about what is the recommended approach if there is a mismatch for a minor version. Is it warning or should that be an error?

I think having this check fills in a very important gap in existing rapids doctor checks, and builds upon information which can be gathered by rapids debug.

[ncclementi]

Yes, I think this would be very valuable. I'm going to document here some of the issues I encounter and useful code snippets that allow me to identify some of the issues.

Fir example, even when there is nor GPU, cudf will install and you'll be able to import it but when trying to do an operation it'll fail.

But the most common issue I encountered Installing either cudf or cuml via pip on an environment with no system cuda. Even though pip will bring a cuda toolkit this caused/causes (maybe not after cupy 14, to be checked)

Now let's say you have a GPU but no system cuda, and you installed cuda via pip. (before cupy 14), you would be able to do some operations, but anything that requires numba jit will fail (I don't recall if it was due to cupy or numba). But at least cupy before cupy 14 required a system cuda.

For example, you'll have scenarios where the first block of code will run but the second will break

#cudf +udf (int)
In [1]: import cudf
  ...: s = cudf.Series([1, 2, 3])
  ...: s.apply(lambda x: x+1)

#cudf +udf (str)
In [1]: import cudf
   ...: s = cudf.Series(['a', 'aa', 'b'])
   ...: s.apply(lambda x: len(x))

Even more, this check helps to see if numba has what it needs when installed as a dependency of cudf

from numba import cuda

@cuda.jit
def kernel():
    pass

print(kernel[1,1]())

For example, once while running this I got

libnvvm.so: cannot open shared object file: No such file or directory

Which lead to discover a bug, because libnvvm was moved/been renamed in cuda 13.1. Related numba PR: NVIDIA/numba-cuda#308

Another example, was a case of cuml when installed via pip/conda, there was a cuda system, but it was updated without me knowing (this happen on a lab machine where the os is controlled by admins), I can see this being a possible issue in some managed systems.

import cupy as cp
X = cp.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1],[3, 2]], cp.float32)
Y = cp.array([1, 1, 1, 2, 2, 2], cp.float32)
from cuml.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(X, Y)

RuntimeError: CuPy failed to load libnvrtc.so.12: OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory

What helped finding the issues, was doing find /usr/local/cuda/ -name "*nvrtc*" and only seeing ....so.13.

Last but not least, sometimes there are issues when installing cudf/cuml and torch. Since torch hard pins some things.

• Here there is a deep dive that James L. did that cover some of the problems for some specific version 'torch' wheels not compatible with 'cuda-toolkit' (CUDA 12 only) build-planning#255 . That being said I don't think we should care much of this case, since there are enough CUDA 12 torch wheels that RAPIDS is compatible with.

Plus PyTorch is now using cuda-toolkit for its dependencies in 2.11+, this will be resolved.

[jacobtomlinson]

Yeah I totally agree with all of this. To answer a few specific questions:

Mismatch on major versions is an automatic error

Yeah this is the bare minimum. I think the most likely scenario for this is the one @ncclementi mentioned where you create an environment and use it for a while, but at some point you upgrade the driver (or it's upgraded for you) and now that environment can't be used on that machine.

I did start looking at this recently and have a WIP draft for a check, I'll push it up. The main thing it needs to finish things off is some actionable advice on what the user should do. If a user has an existing environment and it breaks when they upgrade their drive I want them to be able to run rapids doctor and it tell them something like "Oh you have CUDA Toolkit 12, you need to update your conda environment".

I am curious about what is the recommended approach if there is a mismatch for a minor version

For RAPIDS libraries we follow the minor version compatibility so you should be able to mix minor versions from the same major release.

Overall I think all these things would make great checks, we should add them!

[jacobtomlinson]

Just to add another data point here that we should consider.

Drivers are backward compatible with previous CUDA runtime versions. So if you have a CUDA 13 driver but everything else from the CUDA runtime up the stack is still CUDA 12 then things will work just fine (xref rapidsai/deployment#668).

The problems happen when the runtime is upgraded alongside the driver by upgrading the CTK, but the Python packages are left on an older CUDA version.