Goal: Eliminate CUDA Toolkit dependency for end users

[m96-chan]

Overview

PyGPUkit's long-term goal is to allow end users to run GPU workloads without installing the CUDA Toolkit. Users should only need:

• NVIDIA GPU driver (provides nvcuda.dll / libcuda.so)
• PyGPUkit wheel (ships NVRTC DLL)

This issue tracks progress toward that goal.

---

CUDA Component Dependency Analysis

Component	Build Time	User Runtime	Toolkit Required?	Status
nvcc (CUDA Compiler)	✅ Required	❌ Not needed	Build only	⚠️ Required for wheel build
CUDA Headers (cuda.h, etc.)	✅ Required	❌ Not needed	Build only	⚠️ Required for wheel build
ptxas (PTX Assembler)	✅ Required	❌ Not needed	Build only	⚠️ Required for wheel build
cudart (Runtime Library)	⚙️ Optional	⚙️ Optional	Eliminable	✅ ELIMINATED (driver-only mode)
cuda_driver (nvcuda.dll)	✅ Required	✅ Required	❌ No	✅ OK - provided by GPU driver
NVRTC (nvrtc64_*.dll)	✅ Required	✅ Required	⚠️ Currently yes	🔄 Goal: Ship in wheel

---

Current State

✅ Achieved (User does NOT need Toolkit for these)

Item	How
CUDA Driver API	nvcuda.dll / libcuda.so comes with NVIDIA GPU driver
cudart elimination	PYGPUKIT_DRIVER_ONLY=ON build mode
Driver-only build	PR #48 fixed kernel.hpp compatibility

❌ Not Yet Achieved (User still needs Toolkit)

Item	Blocker	Solution
NVRTC at runtime	nvrtc64_*.dll loaded from CUDA_PATH/bin	Ship NVRTC DLL inside wheel
Windows DLL loading	Needs os.add_dll_directory() for shipped DLLs	Implement in __init__.py

⚠️ Build-Time Only (Acceptable)

Item	Notes
nvcc	Required for compiling .cu files - CI/wheel builders have it
CUDA headers	Required for compilation - CI/wheel builders have it
ptxas	Part of nvcc pipeline - CI/wheel builders have it

---

Implementation Plan

Phase 1: Ship NVRTC in Wheel

• Bundle nvrtc64_120_0.dll (or appropriate version) in wheel
• Add nvrtc64_builtins_*.dll (required companion)
• Update pyproject.toml to include DLLs in package data
• Implement os.add_dll_directory() in __init__.py for Windows

Phase 2: Verify Driver-Only Runtime

• Test on clean Windows install (no CUDA Toolkit, only GPU driver)
• Test on clean Linux install (no CUDA Toolkit, only GPU driver)
• Document minimum GPU driver version requirements

Phase 3: Fallback Handling

• Graceful error when NVRTC missing
• Graceful error when GPU driver too old
• Clear error messages guiding users to install GPU driver (NOT toolkit)

---

Target State

User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only (no CUDA Toolkit)
Wheel ships:   _pygpukit_native.pyd + nvrtc64_*.dll + nvrtc64_builtins_*.dll

---

Related Issues

• Issue PyGPUkit v0.2.3 — Reliability Phase + TF32 TensorCore #42 (v0.2.4 Reliability) - Driver-only mode stabilization
• PR fix(jit): add driver-only mode support for LaunchConfig #48 - Driver-only build fix

Labels

enhancement infrastructure v0.2.4

[m96-chan]

For v0.2.4, the target is to eliminate dependency on the CUDA Toolkit from the user runtime.

[m96-chan]

v0.2.4 Sub-Issues Created

Phase	Issue	Description
Phase 1	#50	Ship NVRTC DLLs in Wheel
Phase 2	#51	Verify Driver-Only Runtime on Clean Systems
Phase 3	#52	Graceful Error Handling for Missing GPU Components

Dependency Order

#50 (Ship NVRTC) → #51 (Verify Runtime) → #52 (Error Handling)

Phase 1 must be completed before Phase 2 testing can begin. Phase 3 can be developed in parallel.

[m96-chan]

PR #57 Progress Update

✅ Completed

Task	Status	Details
Driver-only mode	✅ Done	PYGPUKIT_DRIVER_ONLY flag removed - now the only mode
Dynamic NVRTC loading	✅ Done	NVRTC loaded at runtime, not link-time
Single-binary wheel	✅ Done	Only nvcuda.dll (GPU driver) required
Graceful fallback	✅ Done	Issue #52 closed
Error messages	✅ Done	No "CUDA Toolkit" in user-facing errors

Architecture Change

Original plan was to ship NVRTC in wheel. New approach:

• NVRTC is optional - dynamically loaded only when JIT needed
• Pre-compiled ops work without NVRTC - matmul, add, mul, etc.
• Smaller wheel size - no bundled DLLs

Runtime Dependencies (Final)

DLL	Required	Source
nvcuda.dll	✅ Yes	GPU drivers (user already has)
cudart64_*.dll	❌ No	Eliminated
nvrtc64_*.dll	⚡ Optional	CUDA Toolkit (for JIT only)

Verified with dumpbin

Image has the following dependencies:
    nvcuda.dll        ← Only CUDA dependency!
    python312.dll
    KERNEL32.dll
    ...

Updated Status

Phase 1: Ship NVRTC in Wheel → Changed to Dynamic NVRTC Loading

• Dynamic NVRTC loader (nvrtc_loader.cpp)
• Version-agnostic DLL discovery
• is_nvrtc_available() / get_nvrtc_version() / get_nvrtc_path() API

Phase 2: Driver-Only Runtime ✅

• Test on Windows with GPU driver only
• Benchmark: 17.7 TFLOPS FP32, 28.2 TFLOPS TF32

Phase 3: Fallback Handling ✅ (Issue #52)

• Graceful error when NVRTC missing
• Clear error messages (no "CUDA Toolkit" mention)
• CPU simulation fallback works

Result

User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only
Optional:      CUDA Toolkit (for custom JIT kernels)

Goal achieved! 🎉

[m96-chan]

Goal Achieved! 🎉

PR #57 completes the goal of eliminating CUDA Toolkit dependency for end users.

Final State

User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only (no CUDA Toolkit)
Optional:      CUDA Toolkit (for custom JIT kernels only)

Summary

• Pre-compiled ops (matmul, add, mul, etc.) work with GPU drivers only
• NVRTC is optional - dynamically loaded when JIT is needed
• Single-binary wheel - only nvcuda.dll dependency
• Graceful fallback - clear error messages, CPU simulation available

Thank you for tracking this goal!