enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test wheels against mix of CTK versions by jameslamb · Pull Request #21671 · rapidsai/cudf

jameslamb · 2026-03-05T17:54:55Z

Description

Contributes to rapidsai/build-planning#257

builds CUDA 13 wheels with the 13.0 CTK
ensures wheels ship with a runtime dependency of nvidia-nvjitlink>={whatever-minor-version-they-were-built-against}

Contributes to rapidsai/build-planning#256

updates wheel tests to cover a range of CTK versions (we previously, accidentally, were only testing the latest 12.x and 13.x)

Other changes

ensures conda packages also take on floors of libnvjitlink>={whatever-version-they-were-built-against}

Notes for Reviewers

How I tested this

Used wheels from similar PRs from RAPIDS dependencies, at build and test time:

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

…wheels against mix of CTK versions

copy-pr-bot · 2026-03-05T17:54:59Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

…older-ctk

jameslamb · 2026-03-09T20:37:17Z

~~/merge~~

(edit) Ugh, meant to /ok to test but did this by memory. I've put a DO NOT MERGE label on this so we don't get an unexpected early merge

ci/use_wheels_from_prs.sh

jameslamb · 2026-03-16T12:04:57Z

/ok to test

ci/test_wheel_cudf.sh

bdice · 2026-03-18T18:08:13Z

conda/recipes/libcudf/recipe.yaml

        - ${{ pin_compatible("cuda-version", upper_bound="x", lower_bound="x") }}
        - cuda-nvrtc
-        - libnvjitlink >=${{ cuda_version }}
+        - ${{ pin_compatible("libnvjitlink", lower_bound="x.x.x", upper_bound="x") }}


I think pin_compatible only works correctly if you have that exact package in host. We could verify what rattler says in the build outputs, but I think you might need to list libnvjitlink in host (next to the existing -dev package). The dependency libnvjitlink-dev -> libnvjitlink isn't enough iirc.

Alternatively, wouldn't this be equivalent to us removing the ignore_run_exports on libnvjitlink? The libnvjitlink-dev recipe has:

run_exports: - {{ pin_subpackage("libnvjitlink", max_pin="x") }}

https://github.com/conda-forge/libnvjitlink-feedstock/blob/2fde82c7bc7c3fc354b0f0d808c17dc5036762f1/recipe/meta.yaml#L81-L82

That might be the more correct fix to use here.

we could verify what rattler says

This is resolving correctly to what we want:

│ │ Finalized run dependencies (libcudf-26.04.00a472-cuda13_260318_9e773c7c): ... │ │ │ cuda-version ┆ >=13.1,<14 (RE of [build: cuda-nvcc_linux-64]) │ ... │ │ │ libnvjitlink ┆ >=13.2.51,<14.0a0 (RE of [host: libnvjitlink-dev]) │ ...

I think pin_compatible() just cares about that package making it into the host environment, not whether or not it's explicitly listed in your recipe.

wouldn't this be equivalent to us removing the ignore_run_exports on libnvjitlink?

I think you're right, I was wondering about that. But saw that libnvjitlink-dev run exports were explicitly ignored here:

cudf/conda/recipes/libcudf/recipe.yaml

Line 157 in 03dc96a

- libnvjitlink

And assumed there was some reason for that (like those run exports being wrong for older versions or something).

I'd be happy to be able to remove these and just trust the libnvjitlink run_exports. Let's try that.

Pushed 5fc6e47 doing that, I'll check the rattler-build output before this is merged.

Looks like the run_exports are doing what we want.

CUDA 12.9:

│ │ Finalized run dependencies (libcudf-26.04.00a474-cuda12_260318_0602540d): │ │ ╭────────────────────┬────────────────────────────────────────────────────╮ │ │ │ Name ┆ Spec │ │ │ ╞════════════════════╪════════════════════════════════════════════════════╡ │ │ │ Run dependencies ┆ │ │ │ │ __glibc ┆ >=2.28,<3.0.a0 (RE of [build: sysroot_linux-64]) │ │ │ │ cuda-nvrtc ┆ >=12.9.86,<13.0a0 (RE of [host: cuda-nvrtc-dev]) │ │ │ │ cuda-nvtx ┆ >=12.9.79,<13.0a0 (RE of [host: cuda-nvtx-dev]) │ │ │ │ cuda-version ┆ >=12.9,<13 (RE of [build: cuda-nvcc_linux-64]) │ │ │ │ flatbuffers ┆ >=24.3.25,<24.3.26.0a0 (RE of [host: flatbuffers]) │ │ │ │ libcufile ┆ >=1.14.1.1,<2.0a0 (RE of [host: libcufile-dev]) │ │ │ │ libcurand ┆ >=10.3.10.19,<11.0a0 (RE of [host: libcurand-dev]) │ │ │ │ libgcc ┆ >=14 (RE of [build: gcc_linux-64]) │ │ │ │ ┆ >=14 (RE of [build: gxx_linux-64]) │ │ │ │ libnvcomp ┆ >=5.1.0.21,<6.0a0 (RE of [host: libnvcomp-dev]) │ │ │ │ libnvjitlink ┆ >=12.9.86,<13.0a0 (RE of [host: libnvjitlink-dev]) │ │ │ │ librdkafka ┆ >=2.13.2,<2.14.0a0 (RE of [host: librdkafka]) │ │ │ │ librmm ┆ >=26.4.0a62,<26.5.0a0 (RE of [host: librmm]) │ │ │ │ libstdcxx ┆ >=14 (RE of [build: gxx_linux-64]) │ │ │ │ libzlib ┆ >=1.3.1,<2.0a0 (RE of [host: zlib]) │ │ │ │ ┆ │ │ │ │ Run exports (Weak) ┆ │ │ │ │ libcudf ┆ >=26.4.0a474,<26.5.0a0 │ │ │ ╰────────────────────┴────────────────────────────────────────────────────╯

(build link)

CUDA 13.1:

│ │ Finalized run dependencies (libcudf-26.04.00a474-cuda13_260318_0602540d): │ │ ╭────────────────────┬────────────────────────────────────────────────────╮ │ │ │ Name ┆ Spec │ │ │ ╞════════════════════╪════════════════════════════════════════════════════╡ │ │ │ Run dependencies ┆ │ │ │ │ __glibc ┆ >=2.28,<3.0.a0 (RE of [build: sysroot_linux-64]) │ │ │ │ cuda-nvrtc ┆ >=13.1.115,<14.0a0 (RE of [host: cuda-nvrtc-dev]) │ │ │ │ cuda-nvtx ┆ >=13.1.115,<14.0a0 (RE of [host: cuda-nvtx-dev]) │ │ │ │ cuda-version ┆ >=13.1,<14 (RE of [build: cuda-nvcc_linux-64]) │ │ │ │ flatbuffers ┆ >=24.3.25,<24.3.26.0a0 (RE of [host: flatbuffers]) │ │ │ │ libcufile ┆ >=1.16.1.26,<2.0a0 (RE of [host: libcufile-dev]) │ │ │ │ libcurand ┆ >=10.4.1.81,<11.0a0 (RE of [host: libcurand-dev]) │ │ │ │ libgcc ┆ >=14 (RE of [build: gcc_linux-64]) │ │ │ │ ┆ >=14 (RE of [build: gxx_linux-64]) │ │ │ │ libnvcomp ┆ >=5.1.0.21,<6.0a0 (RE of [host: libnvcomp-dev]) │ │ │ │ libnvjitlink ┆ >=13.2.51,<14.0a0 (RE of [host: libnvjitlink-dev]) │ │ │ │ librdkafka ┆ >=2.13.2,<2.14.0a0 (RE of [host: librdkafka]) │ │ │ │ librmm ┆ >=26.4.0a62,<26.5.0a0 (RE of [host: librmm]) │ │ │ │ libstdcxx ┆ >=14 (RE of [build: gxx_linux-64]) │ │ │ │ libzlib ┆ >=1.3.1,<2.0a0 (RE of [host: zlib]) │ │ │ │ ┆ │ │ │ │ Run exports (Weak) ┆ │ │ │ │ libcudf ┆ >=26.4.0a474,<26.5.0a0 │ │ │ ╰────────────────────┴────────────────────────────────────────────────────╯

(build link)

bdice · 2026-03-18T18:13:05Z

dependencies.yaml

          - matrix:
            packages:
-              - nvidia-nvjitlink>=13.0
+              - nvidia-nvjitlink>=12,<14


I find this a little weird. It's a fake pinning -- none of our packages actually use this -- we just have it to indicate roughly what the package needs. Last I remember, we settled on using CUDA 13 packages here. rapidsai/build-planning#68

Sure, let's use CUDA 13 packages. I'll do that here and in the other PRs in this series.

Pushed 5fc6e47 doing that

…older-ctk

jameslamb · 2026-03-18T20:25:16Z

dependencies.yaml

      table: project.optional-dependencies
      key: test
    includes:
+      - cuda_version


Just picking a place to have a threaded conversation (I don't think this particular line is to blame)... I'm seeing cuML-related conda tests segfaulting:

Fatal Python error: Segmentation fault ... =========================== short test summary info ============================ FAILED python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cuml.py::test_clustering - worker 'gw4' crashed while running 'test_cuml.py::test_clustering' FAILED python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cuml.py::test_pipeline - worker 'gw3' crashed while running 'test_cuml.py::test_pipeline' FAILED python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cuml.py::test_linear_regression - worker 'gw0' crashed while running 'test_cuml.py::test_linear_regression' FAILED python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cuml.py::test_random_forest - worker 'gw2' crashed while running 'test_cuml.py::test_random_forest' FAILED python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_cuml.py::test_logistic_regression - worker 'gw1' crashed while running 'test_cuml.py::test_logistic_regression' ========================= 5 failed, 3 passed in 7.02s ==========================

(third-part-pandas-integration-tests link)

I see conda Python tests segfaulting in recent cuML CI too

e.g. on rapidsai/cuml#7907

https://github.com/rapidsai/cuml/actions/runs/23258577641/job/67624250836

and on rapidsai/cuml#7908

https://github.com/rapidsai/cuml/actions/runs/23249337850/job/67589124096

And in nvforest (which has some code that's also currently in cuML) as well, e.g. on rapidsai/nvforest#88

https://github.com/rapidsai/nvforest/actions/runs/23229511499/job/67622239496

bdice

Cool, cool!

jameslamb · 2026-03-18T21:26:31Z

Merging despite the failing (non-required) third-party-integration-tests-cudf-pandas job. I suspect those failures are unrelated to these changes, especially since I saw similar issues in conda-based jobs with cuML in other repos and PRs (links in #21671 (comment)).

jameslamb · 2026-03-18T21:26:37Z

/merge

…wheels against mix of CTK versions (#5457) Fixes #5443 Contributes to rapidsai/build-planning#257 * builds CUDA 13 wheels with the 13.0 CTK * ensures wheels ship with a runtime dependency of `nvidia-nvjitlink>={whatever-minor-version-they-were-built-against}` Contributes to rapidsai/build-planning#256 * updates wheel tests to cover a range of CTK versions (we previously, accidentally, were only testing the latest 12.x and 13.x) Other changes * ensures conda packages also take on floors of `libnvjitlink>={whatever-version-they-were-built-against}` ## Notes for Reviewers ### How I tested this This uses wheels from similar PRs from RAPIDS dependencies, at build and test time: * rapidsai/cudf#21671 * rapidsai/kvikio#942 * rapidsai/raft#2971 * rapidsai/rmm#2270 * rapidsai/ucxx#604 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #5457

jameslamb added 2 commits March 5, 2026 10:45

enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test …

4843067

…wheels against mix of CTK versions

more wheel-testing script changes

395396e

jameslamb added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Mar 5, 2026

Merge branch 'main' into test-older-ctk

48e4e5c

github-actions bot assigned jameslamb Mar 5, 2026

github-actions bot added Python Affects Python cuDF API. cudf.pandas Issues specific to cudf.pandas labels Mar 5, 2026

github-project-automation bot added this to cuDF Python Mar 5, 2026

GPUtester moved this to In Progress in cuDF Python Mar 5, 2026

This was referenced Mar 5, 2026

wheels: build with oldest CUDA 13 CTK (13.0.x) rapidsai/build-planning#257

Closed

wheels CI: test mix of cuda-toolkit version in CI rapidsai/build-planning#256

Closed

jameslamb and others added 6 commits March 5, 2026 12:58

ugh, left-behind URL

7db1f07

Merge branch 'main' into test-older-ctk

2b01bbf

Merge branch 'test-older-ctk' of github.com:jameslamb/cudf into test-…

876d693

…older-ctk

Merge branch 'main' into test-older-ctk

fa194a9

new commits, remove gha-tools fork

43a3248

Merge branch 'main' into test-older-ctk

0d3a5ea

jameslamb mentioned this pull request Mar 9, 2026

enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test wheels against mix of CTK versions rapidsai/cugraph#5457

Merged

jameslamb and others added 5 commits March 9, 2026 13:10

update raft commit

740a071

Merge branch 'main' into test-older-ctk

fe2b973

use_cuda_wheels=true

0b49084

Merge branch 'main' of github.com:rapidsai/cudf into test-older-ctk

a5990ff

Merge branch 'test-older-ctk' of github.com:jameslamb/cudf into test-…

0f6a2d0

…older-ctk

github-actions bot added the pylibcudf Issues specific to the pylibcudf package label Mar 9, 2026

jameslamb added DO NOT MERGE Hold off on merging; see PR for details labels Mar 9, 2026

jameslamb commented Mar 9, 2026

View reviewed changes

ci/use_wheels_from_prs.sh Outdated Show resolved Hide resolved

stricter libcudf floor

e789455

jameslamb mentioned this pull request Mar 17, 2026

WIP: wheels: test on mix of CTK versions #21808

Closed

3 tasks

jameslamb added 3 commits March 16, 2026 22:51

ensure constraints actually make it into test environments

4532fd4

remove testing and debugging stuff

d2fbb8c

add use_cuda_wheels short-circuit block

de2c879

jameslamb changed the title ~~WIP: [NOT READY FOR REVIEW] enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test wheels against mix of CTK versions~~ enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test wheels against mix of CTK versions Mar 18, 2026

jameslamb removed the DO NOT MERGE Hold off on merging; see PR for details label Mar 18, 2026

Merge branch 'release/26.04' into test-older-ctk

9e773c7

jameslamb marked this pull request as ready for review March 18, 2026 17:15

jameslamb requested review from a team as code owners March 18, 2026 17:15

jameslamb requested a review from KyleFromNVIDIA March 18, 2026 17:15

KyleFromNVIDIA reviewed Mar 18, 2026

View reviewed changes

ci/test_wheel_cudf.sh Show resolved Hide resolved

jameslamb requested a review from KyleFromNVIDIA March 18, 2026 17:21

KyleFromNVIDIA approved these changes Mar 18, 2026

View reviewed changes

bdice reviewed Mar 18, 2026

View reviewed changes

jameslamb added 2 commits March 18, 2026 13:43

just use libnvjitlink run exports, clearer requirement in pyproject.toml

5fc6e47

Merge branch 'test-older-ctk' of github.com:jameslamb/cudf into test-…

0602540

…older-ctk

jameslamb mentioned this pull request Mar 18, 2026

enforce a floor on libnvjitlink, build wheels with CUDA 13.0.x, test wheels against mix of CTK versions rapidsai/cuvs#1862

Merged

jameslamb requested a review from bdice March 18, 2026 19:04

jameslamb commented Mar 18, 2026

View reviewed changes

bdice approved these changes Mar 18, 2026

View reviewed changes

rapids-bot bot merged commit a9064fe into rapidsai:release/26.04 Mar 18, 2026
118 of 119 checks passed

github-project-automation bot moved this from In Progress to Done in cuDF Python Mar 18, 2026

jameslamb deleted the test-older-ctk branch March 18, 2026 21:48

bdice mentioned this pull request Mar 24, 2026

Always build with JIT+LTO rapidsai/cuvs#1923

Merged

Conversation

jameslamb commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes for Reviewers

How I tested this

Checklist

Uh oh!

copy-pr-bot bot commented Mar 5, 2026

Uh oh!

jameslamb commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jameslamb commented Mar 16, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

jameslamb commented Mar 18, 2026

Uh oh!

jameslamb commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jameslamb commented Mar 5, 2026 •

edited

Loading

jameslamb commented Mar 9, 2026 •

edited

Loading