[Issue]: CUDA 13 compatibility: replace deprecated cuda::std includes/usages with std

[skeyyk]

How is this issue impacting you?

Lower performance than expected

Share Your Debug Logs

Description

Building NVSHMEM with CUDA 13 fails because CUDA 13 removed the old cuda::std headers/namespace (e.g. cuda/std/tuple) and the project still includes and references them in multiple places.
Two immediately observed files:

1. include/device_host/nvshmem_tensor.h
2. include/non_abi/device/common/nvshmemi_tile_utils.cuh

Both contain:

#include "cuda/std/tuple"

and there are many usages of cuda::std::... (e.g. cuda::std::tuple, etc.) across the codebase which should be migrated to the standard C++ namespace std::.

Steps to Reproduce the Issue

No response

NVSHMEM Version

3.3.20

Your platform details

No response

Error Message & Behavior

No response

[benhg]

When we build for CUDA 13, we link against CCCL. CCCL (starting with CUDA 13) has include/cccl and that has a cuda/ subdirectory. Also, starting with CUDA 13, CCCL made some changes in their FindCCCL.cmake which gets shipped when you install CCCL.

Can you make sure find_package(CCCL) is succeeding?

[Qizhi697]

I successfully compiled and built nvshmem v3.4.5-0 under CUDA 13.
However, there is an issue: when I set MPI_HOME=/opt/hpcx/ompi, the compile command incorrectly concatenates /opt/hpcx/ompi/include and /opt/nvshmem/perftest/common, resulting in -I/opt/nvshmem/perftest/common/opt/hpcx/ompi/include , which causes mpi.h not found. When I replace it with only ${MPI_CXX_INCLUDE_DIRS} instead of $<BUILD_INTERFACE:${MPI_CXX_INCLUDE_DIRS}>, it works fine.

nvshmem/perftest/common/CMakeLists.txt

Line 24 in 7dd48c9

target_include_directories(nvshmem_perftest_helper PUBLIC $<BUILD_INTERFACE:${MPI_CXX_INCLUDE_DIRS}>)

[benhg]

@Qizhi697 thank you for that. That's an issue I have a fix for internally but I haven't been able to get it released yet.

[skeyyk]

@benhg , thanks for the clarification above.

I have a follow-up issue:

When installing nvidia-nvshmem-cu13==3.4.5 from PyPI and then building another project (deepep) that depends on NVSHMEM, the compiler still fails due to the same deprecated CUDA headers. Specifically, the following files still include the removed CUDA 13 headers:

include/device_host/nvshmem_tensor.h
include/non_abi/device/common/nvshmemi_tile_utils.cuh

Both contain:

#include "cuda/std/tuple"

Since this happens even with the PyPI package nvidia-nvshmem-cu13==3.4.5, it seems that the prebuilt wheel or source distribution may still contain the old cuda::std references rather than using std::tuple.

Could you please confirm if the PyPI-distributed version (3.4.5) has not yet integrated the CCCL migration?
If so, when can we expect a build that fully supports CUDA 13’s CCCL headers and namespace changes?

[benhg]

Thanks for the additional details. I'm not sure exactly what's packaged in nvidia-nvshmem-cu13==3.4.5 wheel, but I can check and get back to you.

[Jumbo0715]

still have this issue... is there any temporary fix i can use ?

[Jumbo0715]

i have temporarily solved this problem by adding include_dirs.extend(['/usr/local/cuda/include/cccl']) after this line, @skeyyk

https://github.com/deepseek-ai/DeepEP/blob/92fe2deaec24bc92ebd9de276daa6ca9ed602ed4/setup.py#L52

[soodoshll]

i have temporarily solved this problem by adding include_dirs.extend(['/usr/local/cuda/include/cccl']) after this line, @skeyyk

https://github.com/deepseek-ai/DeepEP/blob/92fe2deaec24bc92ebd9de276daa6ca9ed602ed4/setup.py#L52

I guess it's more like a DeepEP problem? nvshmem itself can be built.

[skeyyk]

@soodoshll @Jumbo0715 Thanks for sharing the workaround.

I haven't tested it myself, but in theory another temporary workaround could be setting an environment variable to expose the CCCL include directory, e.g.:

export CUDA_INC_PATH=/usr/local/cuda/include/cccl