`cudaErrorNoKernelImageForDevice` on NVIDIA RTX 5070 (Blackwell) with HYPRE solvers

[nicexin24]

System Environment

• GEOSX Version: 1.1.0 (develop, sha1: 2acb3ef19)
• Operating System: Windows Subsystem for Linux (WSL2), Ubuntu 22.04/24.04
• GPU: NVIDIA GeForce RTX 5070 (Blackwell architecture, Compute Capability 10.0 suspected)
• CUDA Toolkit Version: 12.6
• C++ Compiler: GCC 13.3.0 / GCC 14.0
• MPI: Open MPI v4.1.6

Description

I am encountering a cudaErrorNoKernelImageForDevice error when running any simulation or unit test that utilizes the HYPRE linear solver interface. This occurs even after manually setting CUDA_ARCH to sm_90 (Hopper) in the TPL and GEOSX build configurations to ensure compatibility.

The error specifically triggers during kernel dispatch, with the exception message: "after dispatching exclusive_scan kernel: cudaErrorNoKernelImageForDevice". This happens consistently before calls like HYPRE_IJMatrixAddToValues2 in HypreMatrix.cpp:191.

It appears that the compiled binaries for sm_90 are not being correctly recognized or JIT-compiled for the Blackwell (RTX 50-series) hardware within the WSL environment. Since Blackwell is very new, I suspect there might be a mismatch in how RAJA/Thrust kernels are being packaged or a lack of forward compatibility without explicit PTX (virtual architecture) flags.

Question: Does GEOSX currently require specific flags (like 90-virtual or a newer compute capability) to support Blackwell GPUs, or is there a known issue with the exclusive_scan kernel on this new architecture?

Steps to Reproduce

1. Set CUDA_ARCH to sm_90 in thirdPartyLibs/CMakeLists.txt and the host-config file.
2. Build Third Party Libraries (TPL) and GEOSX from scratch.
3. Run the testMatrices unit test: ctest -R testMatrices -V.
4. Observe the crash during the Hypre/MatrixTest suite.

Host-Config (wsl-ubuntu.cmake)

#################################################################################
# Generated host-config - Final Fix for CUDA 12.6 and sm_90 Compatibility
#################################################################################

# 1. Basic Compiler Settings
set(CMAKE_C_COMPILER "/usr/bin/gcc" CACHE PATH "")
set(CMAKE_CXX_COMPILER "/usr/bin/g++" CACHE PATH "")
set(BLT_CXX_STD "c++17" CACHE STRING "")
set(CMAKE_C_FLAGS "-O2 -w -fpermissive" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS "-O3 -w -fpermissive" CACHE STRING "" FORCE)

# 2. MPI Settings
set(ENABLE_MPI ON CACHE BOOL "")
set(MPI_C_COMPILER "/usr/bin/mpicc" CACHE PATH "")
set(MPI_CXX_COMPILER "/usr/bin/mpicxx" CACHE PATH "")
set(MPIEXEC "/usr/bin/mpirun" CACHE PATH "")

# 3. Explicit CUDA Toolkit and Version Configuration
set(ENABLE_CUDA ON CACHE BOOL "" FORCE)
set(CUDA_TOOLKIT_ROOT_DIR "/usr/local/cuda-12.6" CACHE PATH "" FORCE)
set(CMAKE_CUDA_COMPILER "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" CACHE PATH "" FORCE)

set(CUDA_VERSION "12.6" CACHE STRING "" FORCE)
set(CUDA_VERSION_MAJOR "12" CACHE STRING "" FORCE)
set(CUDA_VERSION_MINOR "6" CACHE STRING "" FORCE)
set(HYPRE_CUDA_VERSION "12.6" CACHE STRING "" FORCE)

# GPU Architecture Settings (Using sm_90 for RTX 5070 to ensure TPL compatibility)
set(CMAKE_CUDA_ARCHITECTURES "90" CACHE STRING "" FORCE)
set(CMAKE_CUDA_FLAGS "-restrict --expt-extended-lambda -arch sm_90" CACHE STRING "" FORCE)

# 4. Solver Interfaces
set(GEOS_LA_INTERFACE "Hypre" CACHE STRING "" FORCE)
set(ENABLE_HYPRE_DEVICE "CUDA" CACHE STRING "" FORCE)

# 5. Other Required Libraries
set(BLAS_LIBRARIES "/usr/lib/x86_64-linux-gnu/libblas.so" CACHE STRING "" FORCE)
set(LAPACK_LIBRARIES "/usr/lib/x86_64-linux-gnu/liblapack.so" CACHE STRING "" FORCE)
set(ENABLE_OPENMP OFF CACHE BOOL "" FORCE)

# 6. Third-Party Library (TPL) Path Logic
set(CONFIG_NAME "wsl-ubuntu")
if(NOT ( EXISTS "${GEOS_TPL_DIR}" AND IS_DIRECTORY "${GEOS_TPL_DIR}" ) )
   set(GEOS_TPL_DIR "${CMAKE_SOURCE_DIR}/../../thirdPartyLibs/install-${CONFIG_NAME}-release" CACHE PATH "" FORCE )
endif()

include(${CMAKE_CURRENT_LIST_DIR}/tpls.cmake)

Test Command 1: Unit Test (ctest)

This command runs the standardized linear algebra interface test.

cd ~/rocksim/GEOS/build-wsl-ubuntu-release
ctest -R testMatrices -V --output-on-failure

Key Error Output:

188: [ RUN      ] Hypre/MatrixTest/0.MatrixMatrixOperations
188: unknown file: Failure
188: C++ exception with description "after dispatching exclusive_scan kernel: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device" thrown in the test body.
188: [  FAILED  ] Hypre/MatrixTest/0.MatrixMatrixOperations, where TypeParam = geos::HypreInterface

---

Test Command 2: Integrated Simulation (geosx)

This command attempts to run a basic 3D single-phase flow simulation using the HYPRE solver.

cd ~/rocksim/GEOS/build-wsl-ubuntu-release
./bin/geosx -i ../inputFiles/singlePhaseFlow/3D_10x10x10_compressible_smoke.xml

Key Error Output (Stack Trace):

***** ERROR
***** LOCATION: .../linearAlgebra/interfaces/hypre/HypreUtils.hpp:154
***** Error cause: err != cudaSuccess
***** Rank 0: Previous CUDA errors found: before HYPRE_IJMatrixAddToValues2 (no kernel image is available for execution on the device at .../HypreMatrix.cpp:191)

** StackTrace **
Frame 0: geos::HypreMatrix::create(...)
Frame 1: geos::PhysicsSolverBase::solveNonlinearSystem(...)
Frame 2: geos::PhysicsSolverBase::nonlinearImplicitStep(...)
...

Additional Context

• Manual Source Patch: I had to manually fix a type-casting error in HYPRE (par_amg_setup.c:3155) to compile with GCC 14: a value of type "void *" cannot be used to initialize an entity of type "HYPRE_Solver".
• Storage: The entire TPL and GEOSX project is located on a secondary physical drive (F: drive) mounted via WSL2.

[victorapm]

Hi @nicexin24, GPU-enabled builds of hypre focus on native Linux environments; we (hypre) do not currently support or test on WSL. Hope this helps! Thanks!

[nicexin24]

Hi @victorapm, fair point on the WSL support—I totally get that native Linux is the priority for the team.

However, I suspect this cudaErrorNoKernelImageForDevice might be less about WSL and more about the new Blackwell (RTX 50-series) architecture. Since these cards are brand new, even users on native Linux might run into this soon when trying to execute sm_90 or compute_90 targets.

Before I dive deeper, do you have any quick recommendations on specific CUDA_ARCH flags or JIT settings (like 90-virtual or similar) that you'd suggest for these new chips? I'm happy to do the legwork and test some different build configurations on my end to see what works—it might help the project stay ahead of the Blackwell rollout.

Thanks!

[victorapm]

Hi @nicexin24, the cuda arch for consumer Blackwell is sm_120

[nicexin24]

Hi @victorapm,

Thanks for the confirmation on the sm_120 architecture.

After multiple attempts, I’ve concluded that building the GPU version of GEOSX in WSL2 on a personal workstation is likely unfeasible. Despite allocating 56GB of RAM and 64GB of Swap to the WSL instance, the compilation of heavy kernels (specifically AcousticWaveEquationDG) consistently triggers OOM (Out of Memory) errors and terminates.

It appears the memory peaks required for compiling these complex C++ templates/CUDA objects exceed the resources available on standard consumer hardware. I am moving my GPU builds to a high-memory HPC cluster and will only use the CPU version locally for meshing and setup.

Closing this issue now as the hardware limitation for local GPU builds is clear. Thanks again!

[rrsettgast]

Hello @nicexin24.
Can you elaborate?
Do you have a branch I can look at?
Did you run a parallel build?
You can just avoid building the seismic package for now.
Also, there are things we can do to break up the compilation units into smaller chunks.

[nicexin24]

Hi @rrsettgast, thanks for the suggestions. I’ve decided to move my production work to our HPC cluster (where the GPU build works fine) and keep a CPU version locally for dev/preprocessing. Since the local GPU setup on Blackwell is becoming quite complex, I'll stop further attempts for now, but here is a summary of the issues I encountered for your reference:

• Branch: develop branch.

• Hardware: RTX 5070 (sm_120) + 56GB RAM + 64GB Swap allocated to WSL2.

• RAJA & TPLs: Setting sm_120 initially caused repeated errors in RAJA and other TPLs, which I resolved by ensuring the architecture flags were correctly propagated through the host-config.

• Parallel Build: I tried -j4 and -j2, which crashed almost immediately at the seismic package. Crucially, even with make -j1, the compilation of AcousticWaveEquationDG.cpp.o still triggered an OOM (Out of Memory) event. Below are the real-time memory logs showing that available memory dropped to 85MB despite the massive swap:

  # Memory Monitor Logs:
  Timestamp, Used(MB), Free(MB), Available(MB), Swap_Used(MB)
  2026-02-06 14:44:15, 55853, 729, 387, 65536
  2026-02-06 14:44:20, 56156, 426, 85,  65536 <-- System Terminated
  

• Skipping the module: Following your suggestion, I successfully skipped the seismic package by setting set(GEOS_ENABLE_WAVEPROPAGATION OFF CACHE BOOL "" FORCE) . With this module disabled, I was able to run make -j6 smoothly with memory usage hovering around 6GB-8GB. Although the build encountered a linking error at 99% when building integration tests (e.g., testWavePropagation) due to the missing seismic symbols, the main geosx executable was successfully generated. I verified that bin/geosx exists and the --help command works correctly.

  # Link Error at 99%:
  /usr/bin/ld: ...undefined reference to `typeinfo for geos::AcousticWaveEquationDG'
  collect2: error: ld returned 1 exit status
  
  

• CTest: I ran ctest and got a 91% pass rate (24 tests failed out of 266).

  • Observation: The failures are highly specific. For instance, in testKrylovSolvers, all Trilinos tests passed, but all Hypre tests failed with the following CUDA error:

    C++ exception with description "... cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device" thrown in SetUp().
    
    

  • Analysis: This suggests that while GEOSX itself might be targeting sm_120, some of the linked TPLs (like Hypre) might not be correctly generating or dispatching kernels for the Blackwell architecture on the develop branch.

    91% tests passed, 24 tests failed out of 266
    Total Test time (real) = 341.91 sec
    The following tests FAILED:
            188 - testMatrices (Failed)
            189 - testVectors (Failed)
            190 - testExternalSolvers (Failed)
            191 - testKrylovSolvers (Failed)
            193 - testLAIHelperFunctions (Failed)
            197 - testFlowStatistics (Failed)
            205 - testDofManager (Failed)
            234 - testSolverStats (Failed)
            237 - testTransmissibility (Failed)
            239 - testSinglePhaseMFDPolyhedral (Failed)
            244 - testCompositionalMultiPhaseMFDPolyhedral (Failed)
            254 - testWavePropagation (Not Run)
            255 - testWavePropagationDG (Not Run)
            256 - testWavePropagationTaper (Not Run)
            257 - testWavePropagationQ2 (Not Run)
            258 - testWavePropagationElasticFirstOrder (Not Run)
            259 - testWavePropagationDAS (Not Run)
            260 - testWavePropagationElasticVTI (Not Run)
            261 - testWavePropagationElasticTTI (Not Run)
            262 - testWavePropagationAttenuationElastic (Not Run)
            263 - testWavePropagationAttenuationAcoustic (Not Run)
            264 - testWavePropagationAttenuationAcousticVTI (Not Run)
            265 - testWavePropagationAcousticFirstOrder (Not Run)
            266 - testWavePropagationAdjoint1 (Not Run)
    Errors while running CTest
    

It seems the current GEOSX GPU kernels (especially for new architectures like Blackwell) are a bit too heavy for standard consumer PC/WSL2 environments. The plan to break up compilation units will definitely help bridge this gap for workstation users!

Thanks again for the help.