Skip to content

Commit 4688a1a

Browse files
jw3126jw3126
authored
Merge pull request #32 from GunnarFarneback/cuda_package_extension
Support CUDA on Julia 1.9+ via a package extension.
2 parents dfb67d6 + bd6a35b commit 4688a1a

File tree

14 files changed

+299
-39
lines changed

14 files changed

+299
-39
lines changed

‎.github/workflows/CI.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ jobs:
1010
fail-fast: false
1111
matrix:
1212
version:
13-
- '1.6'
13+
- '1.9'
1414
- '1'
1515
- 'nightly'
1616
os:

‎Project.toml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name = "ONNXRunTime"
22
uuid = "e034b28e-924e-41b2-b98f-d2bbeb830c6a"
33
authors = ["Jan Weidner <jw3126@gmail.com> and contributors"]
4-
version = "0.3.3"
4+
version = "0.4.0"
55

66
[deps]
77
ArgCheck = "dce04be8-c92d-5529-be00-80e4d2c0e197"
@@ -11,18 +11,27 @@ DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
1111
LazyArtifacts = "4af54fe1-eca0-43a8-85a7-787d91b784e3"
1212
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
1313
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
14-
Requires = "ae029012-a4dd-5104-9daa-d747884805df"
1514

1615
[compat]
1716
ArgCheck = "2"
1817
CEnum = "0.4"
18+
CUDA = "4, 5"
1919
DataStructures = "0.18"
2020
DocStringExtensions = "0.8, 0.9"
21-
Requires = "1"
22-
julia = "1.6"
21+
cuDNN = "1.1"
22+
julia = "1.9"
23+
24+
[extensions]
25+
CUDAExt = ["CUDA", "cuDNN"]
2326

2427
[extras]
2528
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
29+
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
30+
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"
2631

2732
[targets]
28-
test = ["Test"]
33+
test = ["Test", "CUDA", "cuDNN"]
34+
35+
[weakdeps]
36+
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
37+
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"

‎README.md‎

Lines changed: 65 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,24 @@ julia> model(input)
2828
Dict{String, Matrix{Float32}} with 1 entry:
2929
"output" => [2.68127 2.18192 0.525979; -0.135185 2.02199 3.75168]
3030
```
31-
For GPU usage simply do:
31+
32+
For GPU usage the CUDA and cuDNN packages are required and the CUDA
33+
runtime needs to be set to 11.8 or a later 11.x version. To set this
34+
up, do
35+
3236
```julia
33-
pkg> add CUDA
37+
pkg> add CUDA cuDNN
3438

3539
julia> import CUDA
3640

41+
julia> CUDA.set_runtime_version!(v"11.8")
42+
```
43+
44+
Then GPU inference is simply
45+
46+
```julia
47+
julia> import CUDA, cuDNN
48+
3749
julia> ORT.load_inference(path, execution_provider=:cuda)
3850
```
3951

@@ -63,3 +75,54 @@ output_array = GetTensorMutableData(api, output_tensor);
6375
* Use the onnxruntime python bindings via [PyCall.jl](https://github.com/JuliaPy/PyCall.jl).
6476
* [ONNX.jl](https://github.com/FluxML/ONNX.jl)
6577
* [ONNXNaiveNASflux.jl](https://github.com/DrChainsaw/ONNXNaiveNASflux.jl)
78+
79+
# Breaking Changes in version 0.4.
80+
81+
* Support for CUDA.jl is changed from version 3 to versions 4 and 5.
82+
83+
* Support for Julia versions less than 1.9 is dropped. The reason for
84+
this is to switch the conditional support of GPUs from being based
85+
on the Requires package to being a package extension. As a
86+
consequence the ONNXRunTime GPU support can now be precompiled and
87+
the CUDA.jl versions can be properly controlled via Compat.
88+
89+
# Setting the CUDA Runtime Version in Tests
90+
91+
For GPU tests using ONNXRunTime, naturally the tests must depend on
92+
and import CUDA and cuDNN. Additionally a supported CUDA runtime
93+
version needs to be used, which can be somewhat tricky to set up for
94+
the tests.
95+
96+
First some background. What `CUDA.set_runtime_version!(v"11.8")`
97+
effectively does is to
98+
99+
1. Add a `LocalPreferences.toml` file containing
100+
101+
```
102+
[CUDA_Runtime_jll]
103+
version = "11.8"
104+
```
105+
106+
2. In `Project.toml`, add
107+
```
108+
[extras]
109+
CUDA_Runtime_jll = "76a88914-d11a-5bdc-97e0-2f5a05c973a2"
110+
```
111+
112+
If your test environment is defined by a `test` target in the top
113+
`Project.toml` you need to
114+
115+
1. Add a `LocalPreferences.toml` in your top directory with the same
116+
contents as above.
117+
118+
2. Add `CUDA_Runtime_jll` to the `extras` section of `Project.toml`.
119+
120+
3. Add `CUDA_Runtime_jll` to the `test` target of `Project.toml`.
121+
122+
If your test environment is defined by a `Project.toml` in the `test`
123+
directory, you instead need to
124+
125+
1. Add a `test/LocalPreferences.toml` file with the same contents as
126+
above.
127+
128+
2. Add `CUDA_Runtime_jll` to the `extras` section of `test/Project.toml`.

‎ext/CUDAExt.jl‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
module CUDAExt
2+
3+
# These functions are only defined for diagnostic purposes. Otherwise
4+
# the CUDA extension only relies on the CUDA and cuDNN dependencies to
5+
# have loaded the libraries needed by ONNXRunTime's CUDA execution
6+
# provider.
7+
import CUDA
8+
cuda_functional() = CUDA.functional()
9+
cuda_runtime_version() = CUDA.runtime_version()
10+
11+
end

‎src/ONNXRunTime.jl‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
module ONNXRunTime
2-
using Requires:@require
32

43
function _perm(arr::AbstractArray{T,N}) where {T,N}
54
ntuple(i->N+1-i, N)
@@ -11,13 +10,8 @@ function reversedims_lazy(arr)
1110
PermutedDimsArray(arr, _perm(arr))
1211
end
1312

13+
include("versions.jl")
1414
include("capi.jl")
1515
include("highlevel.jl")
1616

17-
function __init__()
18-
@require CUDA="052768ef-5323-5732-b1bb-66c8b64840ba" begin
19-
CUDA.functional() && include("cuda.jl")
20-
end
21-
end
22-
2317
end #module

‎src/cuda.jl‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎src/highlevel.jl‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,22 @@ function load_inference(path::AbstractString; execution_provider::Symbol=:cpu,
6565
if execution_provider === :cpu
6666
session_options = CreateSessionOptions(api)
6767
elseif execution_provider === :cuda
68-
if !(isdefined(@__MODULE__, :CUDA))
69-
@warn """
70-
The $(repr(execution_provider)) requires the CUDA.jl package to be available. Try adding `import CUDA` to your code.
71-
"""
68+
CUDAExt = Base.get_extension(@__MODULE__, :CUDAExt)
69+
if isnothing(CUDAExt)
70+
error("""
71+
The $(repr(execution_provider)) execution provider requires the CUDA.jl and cuDNN.jl packages to be available. Try adding `import CUDA, cuDNN` to your code.
72+
""")
73+
elseif !getfield(CUDAExt, :cuda_functional)()
74+
error("""
75+
The $(repr(execution_provider)) execution provider requires CUDA to be functional. See `CUDA.functional`.
76+
""")
77+
else
78+
cuda_runtime_version = getfield(CUDAExt, :cuda_runtime_version)()
79+
if !(cuda_runtime_supported_version <= cuda_runtime_version < cuda_runtime_upper_bound)
80+
error("""
81+
Found CUDA runtime version $(cuda_runtime_version). The $(repr(execution_provider)) execution provider requires a CUDA runtime version of at least $(cuda_runtime_supported_version) but less than $(cuda_runtime_upper_bound). See `CUDA.set_runtime_version!` and the package README.
82+
""")
83+
end
7284
end
7385
session_options = CreateSessionOptions(api)
7486
cuda_options = OrtCUDAProviderOptions()

‎src/versions.jl‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Version number of the ONNXRunTime library and supported versions of
2+
# the CUDA runtime for GPU processing with the CUDA execution
3+
# provider.
4+
#
5+
# * `onnxruntime_version`: This number must match the version number
6+
# reported by the ONNXRunTime library, which is verified in the
7+
# tests. The only real purpose of this variable is to help keep the
8+
# next one up to date when the library is updated.
9+
#
10+
# * `cuda_runtime_supported_version`: This is the lowest supported
11+
# version of the ONNX runtime library, which should match the
12+
# information from
13+
# https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
14+
#
15+
# * `cuda_runtime_upper_bound`: The lowest CUDA runtime version which
16+
# is *not* accepted. Presumably CUDA runtime follows semantic
17+
# versioning so this can automatically be set to the next major
18+
# version.
19+
const onnxruntime_version = v"1.15.1"
20+
const cuda_runtime_supported_version = v"11.8"
21+
const cuda_runtime_upper_bound = VersionNumber(cuda_runtime_supported_version.major + 1)

‎test/LocalPreferences.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[CUDA_Runtime_jll]
2+
version = "11.8"

‎test/Project.toml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
[deps]
22
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
33
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
4+
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"
45

56
[compat]
6-
CUDA = "3"
7+
CUDA = "5"
8+
cuDNN = "1.2"
9+
10+
[extras]
11+
CUDA_Runtime_jll = "76a88914-d11a-5bdc-97e0-2f5a05c973a2"

0 commit comments

Comments
 (0)