Skip to content

WIP: wheels: test on mix of CTK versions#21808

Closed
jameslamb wants to merge 5 commits intorapidsai:release/26.04from
jameslamb:no-builds-test-older-ctk
Closed

WIP: wheels: test on mix of CTK versions#21808
jameslamb wants to merge 5 commits intorapidsai:release/26.04from
jameslamb:no-builds-test-older-ctk

Conversation

@jameslamb
Copy link
Copy Markdown
Member

@jameslamb jameslamb commented Mar 17, 2026

Description

Similar to #21671, but without the changes to building wheels with CTK 13.0. Using this to test whether cuDF wheels exhibit similar issues to rapidsai/cugraph#5443 in a runtime environment with an older nvidia-nvjitlink.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@jameslamb jameslamb added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Mar 17, 2026
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot bot commented Mar 17, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added Python Affects Python cuDF API. cudf.pandas Issues specific to cudf.pandas pylibcudf Issues specific to the pylibcudf package labels Mar 17, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python Mar 17, 2026
@jameslamb
Copy link
Copy Markdown
Member Author

/ok to test

use_cuda_wheels: "true"
packages:
- cuda-toolkit==13.0.*
- nvidia-nvjitlink==13.0.*
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aha!

libcudf wheels currently have a requirement like nvidia-nvjitlink>=13.0

- nvidia-nvjitlink>=13.0

That floor is wrong... it should be >=13.1 because we build against CTK 13.1 (as I've been testing in rapidsai/build-planning#256 and rapidsai/build-planning#257).

This nvidia-nvjitlink==13.0.* pin in constraints forced out the runtime issues users would see if they installed cudf-cu13 and nvidia-nvjitlink==13.0.* together (as might happen in an environment with torch wheels):

FAILED test_transform.py::test_transform_udf - RuntimeError: Jitify fatal error: Dynamically loaded libnvJitLink.so.13 is too old; it is version 13.0, need at least version 13.1
FAILED io/test_parquet.py::test_read_parquet_filters_jit[a.txt-pa_filter0-plc_filter0-True] - RuntimeError: Jitify fatal error: Dynamically loaded libnvJitLink.so.13 is too old; it is version 13.0, need at least version 13.1
FAILED io/test_parquet.py::test_read_parquet_filters_jit[a.txt-pa_filter1-plc_filter1-True] - RuntimeError: Jitify fatal error: Dynamically loaded libnvJitLink.so.13 is too old; it is version 13.0, need at least version 13.1
FAILED io/test_parquet.py::test_read_parquet_filters_jit[binary_source_or_sink1-pa_filter0-plc_filter0-True] - RuntimeError: Jitify fatal error: Uninitialized
FAILED io/test_parquet.py::test_read_parquet_filters_jit[binary_source_or_sink1-pa_filter1-plc_filter1-True] - RuntimeError: Jitify fatal error: Uninitialized
FAILED io/test_parquet.py::test_read_parquet_filters_jit[BytesIO-pa_filter0-plc_filter0-True] - RuntimeError: Jitify fatal error: Uninitialized
FAILED io/test_parquet.py::test_read_parquet_filters_jit[BytesIO-pa_filter1-plc_filter1-True] - RuntimeError: Jitify fatal error: Uninitialized

(build link)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And notice this only happens on the CUDA 13.0 jobs (not 13.1):

Image

@jameslamb
Copy link
Copy Markdown
Member Author

This was just for testing, to get the error messages in #21808 (comment) confirming that our nvidia-nvjitlink wheel pins here were too permissive.

Closing this, see #21671

@jameslamb jameslamb closed this Mar 18, 2026
@github-project-automation github-project-automation bot moved this from In Progress to Done in cuDF Python Mar 18, 2026
@jameslamb jameslamb deleted the no-builds-test-older-ctk branch March 18, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cudf.pandas Issues specific to cudf.pandas improvement Improvement / enhancement to an existing function non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants