Skip to content

Commit 3fe4420

Browse files
johncalespjohncalesp
authored
Include CUPTI version in wheels (#44)
* This PR's main topic is adding the CUPTI version to the build process for versions 11.3 and 11.6 we don't have nvidia-cupti version that match * changed env name from version_tag to version_cuda_tag * added env variable version_cuda_tag to whl-build-ec2 for the release to pypi * added new env variable for pytorch version * added env var to build-ec2 * added new bash script to update the version for new releases to replace increment changed setup script to use pip setup instead of python build wheels updated yaml files to reflect changes in setup.py updated version in __init__.py to get the information from importlib commented whl-build-ec2 for initial testing * removed python3.7 from build because importlib is not found for that version removed start and stop runner because we only need to start-up the ec2 instance updated version in README.md --------- Co-authored-by: John Calderon <john.calderon@centml.ai>
1 parent c8edebe commit 3fe4420

File tree

10 files changed

+139
-136
lines changed

10 files changed

+139
-136
lines changed

‎.github/workflows/whl-build-all.yaml‎

Lines changed: 77 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ jobs:
99
runs-on: [self-hosted]
1010
steps:
1111
- name: Configure AWS Credentials
12-
uses: aws-actions/configure-aws-credentials@v1
12+
uses: aws-actions/configure-aws-credentials@v4
1313
with:
1414
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
1515
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -27,7 +27,7 @@ jobs:
2727
runs-on: [self-hosted, cu113]
2828
steps:
2929
- name: Configure AWS Credentials
30-
uses: aws-actions/configure-aws-credentials@v1
30+
uses: aws-actions/configure-aws-credentials@v4
3131
with:
3232
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
3333
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -42,28 +42,36 @@ jobs:
4242
ls -la
4343
4444
- name: Fetch repository
45-
uses: actions/checkout@v3
46-
47-
- name: Build Python3.7 wheel
48-
run: |
49-
./build_scripts/build_wheel.sh python3.7
45+
uses: actions/checkout@v4
5046

5147
- name: Build Python3.8 wheel
5248
run: |
5349
./build_scripts/build_wheel.sh python3.8
50+
env:
51+
CUDA_TAG: cu113
52+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
5453

5554
- name: Build Python3.9 wheel
5655
run: |
5756
./build_scripts/build_wheel.sh python3.9
58-
57+
env:
58+
CUDA_TAG: cu113
59+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
60+
5961
- name: Build Python3.10 wheel
6062
run: |
6163
./build_scripts/build_wheel.sh python3.10
62-
64+
env:
65+
CUDA_TAG: cu113
66+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
67+
6368
- name: Build Python3.11 wheel
6469
run: |
6570
./build_scripts/build_wheel.sh python3.11
66-
71+
env:
72+
CUDA_TAG: cu113
73+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
74+
6775
- name: Upload wheels to S3
6876
run: |
6977
aws s3 cp analyzer/dist/ s3://centml-releases/habitat/wheels-cu113/ --recursive --exclude "*" --include "*.whl"
@@ -82,7 +90,7 @@ jobs:
8290
runs-on: [self-hosted, cu116]
8391
steps:
8492
- name: Configure AWS Credentials
85-
uses: aws-actions/configure-aws-credentials@v1
93+
uses: aws-actions/configure-aws-credentials@v4
8694
with:
8795
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
8896
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -97,33 +105,35 @@ jobs:
97105
ls -la
98106
99107
- name: Fetch repository
100-
uses: actions/checkout@v3
101-
102-
- name: Build Python3.7 wheel
103-
run: |
104-
./build_scripts/build_wheel.sh python3.7
108+
uses: actions/checkout@v4
105109

106110
- name: Build Python3.8 wheel
107111
run: |
108112
./build_scripts/build_wheel.sh python3.8
109-
113+
env:
114+
CUDA_TAG: cu116
115+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
116+
110117
- name: Build Python3.9 wheel
111118
run: |
112119
./build_scripts/build_wheel.sh python3.9
113-
120+
env:
121+
CUDA_TAG: cu116
122+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
123+
114124
- name: Build Python3.10 wheel
115125
run: |
116126
./build_scripts/build_wheel.sh python3.10
117-
127+
env:
128+
CUDA_TAG: cu116
129+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
130+
118131
- name: Build Python3.11 wheel
119132
run: |
120133
./build_scripts/build_wheel.sh python3.11
121-
122-
- name: Upload wheel artifact
123-
uses: actions/upload-artifact@v3
124-
with:
125-
name: wheels-cu116
126-
path: analyzer/dist/*.whl
134+
env:
135+
CUDA_TAG: cu116
136+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
127137

128138
- name: Upload wheels to S3
129139
run: |
@@ -143,7 +153,7 @@ jobs:
143153
runs-on: [self-hosted, cu117]
144154
steps:
145155
- name: Configure AWS Credentials
146-
uses: aws-actions/configure-aws-credentials@v1
156+
uses: aws-actions/configure-aws-credentials@v4
147157
with:
148158
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
149159
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -158,27 +168,35 @@ jobs:
158168
ls -la
159169
160170
- name: Fetch repository
161-
uses: actions/checkout@v3
162-
163-
- name: Build Python3.7 wheel
164-
run: |
165-
./build_scripts/build_wheel.sh python3.7
166-
171+
uses: actions/checkout@v4
172+
167173
- name: Build Python3.8 wheel
168174
run: |
169175
./build_scripts/build_wheel.sh python3.8
176+
env:
177+
CUDA_TAG: cu117
178+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
170179

171180
- name: Build Python3.9 wheel
172181
run: |
173182
./build_scripts/build_wheel.sh python3.9
183+
env:
184+
CUDA_TAG: cu117
185+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
174186

175187
- name: Build Python3.10 wheel
176188
run: |
177189
./build_scripts/build_wheel.sh python3.10
178-
190+
env:
191+
CUDA_TAG: cu117
192+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
193+
179194
- name: Build Python3.11 wheel
180195
run: |
181196
./build_scripts/build_wheel.sh python3.11
197+
env:
198+
CUDA_TAG: cu117
199+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.7.101,nvidia-cuda-runtime-cu11==11.7.99
182200

183201
- name: Upload wheels to S3
184202
run: |
@@ -198,7 +216,7 @@ jobs:
198216
runs-on: [self-hosted, cu118]
199217
steps:
200218
- name: Configure AWS Credentials
201-
uses: aws-actions/configure-aws-credentials@v1
219+
uses: aws-actions/configure-aws-credentials@v4
202220
with:
203221
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
204222
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -213,27 +231,35 @@ jobs:
213231
ls -la
214232
215233
- name: Fetch repository
216-
uses: actions/checkout@v3
217-
218-
- name: Build Python3.7 wheel
219-
run: |
220-
./build_scripts/build_wheel.sh python3.7
234+
uses: actions/checkout@v4
221235

222236
- name: Build Python3.8 wheel
223237
run: |
224238
./build_scripts/build_wheel.sh python3.8
239+
env:
240+
CUDA_TAG: cu118
241+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.8.87,nvidia-cuda-runtime-cu11==11.8.89
225242

226243
- name: Build Python3.9 wheel
227244
run: |
228245
./build_scripts/build_wheel.sh python3.9
246+
env:
247+
CUDA_TAG: cu118
248+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.8.87,nvidia-cuda-runtime-cu11==11.8.89
229249

230250
- name: Build Python3.10 wheel
231251
run: |
232252
./build_scripts/build_wheel.sh python3.10
253+
env:
254+
CUDA_TAG: cu118
255+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.8.87,nvidia-cuda-runtime-cu11==11.8.89
233256

234257
- name: Build Python3.11 wheel
235258
run: |
236259
./build_scripts/build_wheel.sh python3.11
260+
env:
261+
CUDA_TAG: cu118
262+
EXTRA_REQUIRES: nvidia-cuda-cupti-cu11==11.8.87,nvidia-cuda-runtime-cu11==11.8.89
237263

238264
- name: Upload wheels to S3
239265
run: |
@@ -253,7 +279,7 @@ jobs:
253279
runs-on: [self-hosted, cu121]
254280
steps:
255281
- name: Configure AWS Credentials
256-
uses: aws-actions/configure-aws-credentials@v1
282+
uses: aws-actions/configure-aws-credentials@v4
257283
with:
258284
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
259285
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
@@ -268,28 +294,32 @@ jobs:
268294
ls -la
269295
270296
- name: Fetch repository
271-
uses: actions/checkout@v3
272-
273-
- name: Build Python3.7 wheel
274-
run: |
275-
./build_scripts/build_wheel.sh python3.7
297+
uses: actions/checkout@v4
276298

277299
- name: Build Python3.8 wheel
278300
run: |
279301
./build_scripts/build_wheel.sh python3.8
302+
env:
303+
CUDA_TAG: cu121
280304

281305
- name: Build Python3.9 wheel
282306
run: |
283307
./build_scripts/build_wheel.sh python3.9
308+
env:
309+
CUDA_TAG: cu121
284310

285311
- name: Build Python3.10 wheel
286312
run: |
287313
./build_scripts/build_wheel.sh python3.10
288-
314+
env:
315+
CUDA_TAG: cu121
316+
289317
- name: Build Python3.11 wheel
290318
run: |
291319
./build_scripts/build_wheel.sh python3.11
292-
320+
env:
321+
CUDA_TAG: cu121
322+
293323
- name: Upload wheels to S3
294324
run: |
295325
aws s3 cp analyzer/dist/ s3://centml-releases/habitat/wheels-cu121/ --recursive --exclude "*" --include "*.whl"

‎.github/workflows/whl-build-ec2.yaml‎

Lines changed: 14 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,8 @@ jobs:
2929
3030
- name: Update version
3131
run: |
32-
pip install incremental
3332
pushd analyzer
34-
python3 -m incremental.update habitat --newversion=${{ github.event.inputs.tag }}
33+
bash git_update_version.sh ${{ github.event.inputs.tag }}
3534
popd
3635
3736
- name: Commit updated version number and tag it
@@ -43,89 +42,42 @@ jobs:
4342
git push -u origin release/${{ github.event.inputs.tag }}
4443
git tag ${{ github.event.inputs.tag }}
4544
46-
start-runner:
47-
name: Start self-hosted EC2 runner
48-
runs-on: ubuntu-latest
49-
needs:
50-
- create-release-branch
51-
outputs:
52-
label: ${{ steps.start-ec2-runner.outputs.label }}
53-
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
54-
steps:
55-
- name: Configure AWS credentials
56-
uses: aws-actions/configure-aws-credentials@v1
57-
with:
58-
aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
59-
aws-region: us-east-1
60-
aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}
61-
- name: Start EC2 runner
62-
id: start-ec2-runner
63-
uses: CentML/GPUClusterSwitchGithubAction@v1.3.10
64-
with:
65-
mode: start
66-
github-token: ${{ secrets.CENTML_PAT }}
67-
t4: true
68-
a10g: false
69-
v100: false
70-
7145
build-release:
72-
runs-on: [self-hosted, cu117]
73-
needs:
74-
- start-runner
46+
runs-on: [self-hosted, cu121]
7547
steps:
7648
- name: Fetch repository
7749
uses: actions/checkout@v3
7850
with:
7951
ref: release/${{ github.event.inputs.tag }}
8052

81-
- name: Build Python3.7 distribution
82-
run: |
83-
./build_scripts/build_wheel.sh python3.7
84-
8553
- name: Build Python3.8 wheel
8654
run: |
8755
./build_scripts/build_wheel.sh python3.8
88-
56+
env:
57+
CUDA_TAG: cu121
58+
8959
- name: Build Python3.9 wheel
9060
run: |
9161
./build_scripts/build_wheel.sh python3.9
92-
62+
env:
63+
CUDA_TAG: cu121
64+
9365
- name: Build Python3.10 wheel
9466
run: |
9567
./build_scripts/build_wheel.sh python3.10
96-
68+
env:
69+
CUDA_TAG: cu121
70+
9771
- name: Build Python3.11 wheel
9872
run: |
9973
./build_scripts/build_wheel.sh python3.11
100-
74+
env:
75+
CUDA_TAG: cu121
76+
10177
- uses: actions/upload-artifact@v3
10278
with:
10379
name: ${{ github.event.inputs.tag }}
10480
path: analyzer/dist/*.whl
105-
106-
stop-runner:
107-
name: Stop self-hosted EC2 runner
108-
needs:
109-
- start-runner # required to get output from the start-runner job
110-
- build-release # required to wait when the main job is done
111-
runs-on: ubuntu-latest
112-
if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
113-
steps:
114-
- name: Configure AWS credentials
115-
uses: aws-actions/configure-aws-credentials@v1
116-
with:
117-
aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
118-
aws-region: us-east-1
119-
aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}
120-
- name: Stop EC2 runner
121-
id: stop-ec2-runner
122-
uses: CentML/GPUClusterSwitchGithubAction@v1.3.10
123-
with:
124-
mode: stop
125-
github-token: ${{ secrets.CENTML_PAT }}
126-
t4: true # required to match the list above - otherwise the runners will not stop
127-
a10g: false # required to match the list above - otherwise the runners will not stop
128-
v100: false
12981

13082
publish-to-github:
13183
name: Publish to Github

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ DeepView.Predict is a tool that predicts a deep neural network's training iterat
2222
<h2 id="installation">Installation</h2>
2323

2424
To run DeepView.Predict, you need:
25-
- [Python 3.6+](https://www.python.org/)
26-
- [Pytorch 1.1.0+](https://pytorch.org/)
25+
- [Python 3.8+](https://www.python.org/)
26+
- [Pytorch 1.13.1+](https://pytorch.org/)
2727
- A system equiped with an Nvidia GPU with properly configured CUDA
2828

2929
Currently, we have predictors for the following Nvidia GPUs:

0 commit comments

Comments
 (0)