Skip to content

GPU/CPU Management & Inference #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
umama-rahman1 merged 13 commits into from
Jan 12, 2025
Merged

GPU/CPU Management & Inference #19

umama-rahman1 merged 13 commits into from
Jan 12, 2025

Conversation

@umama-rahman1
Copy link
Contributor

@umama-rahman1 umama-rahman1 commented Dec 16, 2024 •
edited
Loading

Related Task

Changes

  • Added GPU machine creation
  • Script to run on creating machine and expose port for Ollama inference
  • Ollama inference url getter
  • Pull model onto GPU machine
  • Delete model from GPU machine

Documentation

  • New /GPU machine endpoints:
image

Local Testing

Create GPU Machine:

image image

Script run on creating machine and to expose ollama port:
image

Pull Model to machine:

image image

Delete Model from machine:

image image

Get Inference URL for machine access:

image image

Inference:

image

All endpoints refactored:

image

@umama-rahman1 umama-rahman1 added the dev Code Development work label Dec 16, 2024
@umama-rahman1 umama-rahman1 self-assigned this Dec 16, 2024
@umama-rahman1 umama-rahman1 changed the title GPU Management GPU Management & Ollama inference Dec 16, 2024
@umama-rahman1 umama-rahman1 changed the title GPU Management & Ollama inference GPU/CPU Management & Ollama inference Dec 16, 2024
@umama-rahman1 umama-rahman1 requested review from Daayim, K-rolls and cmatthews20 and removed request for Daayim and K-rolls December 16, 2024 02:45
@K-rolls K-rolls requested review from K-rolls and cmatthews20 and removed request for K-rolls and cmatthews20 December 17, 2024 18:38
K-rolls
K-rolls approved these changes Dec 17, 2024
View reviewed changes
Copy link
Contributor

@K-rolls K-rolls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small changes requested, great work buddy

Daayim
Daayim requested changes Jan 10, 2025
View reviewed changes
Copy link
Contributor

@Daayim Daayim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff. I just had a few questions if you don't mind taking a look.

@cmatthews20 cmatthews20 changed the title GPU/CPU Management & Ollama inference GPU/CPU Management & Inference Jan 11, 2025
K-rolls
K-rolls approved these changes Jan 11, 2025
View reviewed changes
Copy link
Contributor

@K-rolls K-rolls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Daayim
Daayim approved these changes Jan 12, 2025
View reviewed changes
Copy link
Contributor

@Daayim Daayim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@umama-rahman1 umama-rahman1 merged commit c65b54c into main Jan 12, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@K-rolls K-rolls K-rolls approved these changes

@cmatthews20 cmatthews20 cmatthews20 approved these changes

@Daayim Daayim Daayim approved these changes

Assignees

@umama-rahman1 umama-rahman1

Labels

dev Code Development work

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@umama-rahman1 @K-rolls @cmatthews20 @Daayim