Skip to content

Commit 0dae95a

Browse files
committed
Get rid of old Llama3 vision version
1 parent 11d1af7 commit 0dae95a

File tree

7 files changed

+6
-95
lines changed

7 files changed

+6
-95
lines changed

README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,9 @@ docker compose up rest_api
1919
* The open-source implementation are based on the `transformers` library. I have experimented with `vLLM`, but it made the GPU run OOM. More fiddling is needed.
2020
* I have extracted a REST API using `FastAPI` to decouple the frontend streamlit code from the inference server.
2121
* The app supports small open-source models atm, because the inference server is running a single 24GB VRAM GPU. We will hopefully scale this backend up soon.
22+
23+
## Archive: Installation Tips
24+
Installation for the quantized model in `llama_cpp`:
25+
```shell
26+
CMAKE_ARGS="-DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.5 -DCUDAToolkit_ROOT=/usr/local/cuda-12.5 -DCUDAToolkit_INCLUDE_DIR=/usr/local/cuda-12/include -DCUDAToolkit_LIBRARY_DIR=/usr/local/cuda-12.5/lib64" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir
27+
```

llmlib/llmlib/llama3/.gitignore

Lines changed: 0 additions & 1 deletion
This file was deleted.

llmlib/llmlib/llama3/README.md

Lines changed: 0 additions & 5 deletions
This file was deleted.

llmlib/llmlib/llama3/__init__.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

llmlib/llmlib/llama3/llama3_vision_8b.py

Lines changed: 0 additions & 67 deletions
This file was deleted.

llmlib/llmlib/runtime.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
from .gemini.gemini_code import GeminiAPI
33
from .gemma import PaliGemma2
44
from .minicpm import MiniCPM
5-
from .llama3 import LLama3Vision8B
65
from .model_registry import ModelEntry, ModelRegistry, model_entries_from_mult_ids
76
from .openai.openai_completion import OpenAIModel
87
from .phi3.phi3 import Phi3Vision
@@ -14,7 +13,6 @@ def filled_model_registry() -> ModelRegistry:
1413
*model_entries_from_mult_ids(MiniCPM),
1514
ModelEntry.from_cls_with_id(Apollo7B),
1615
ModelEntry.from_cls_with_id(Phi3Vision),
17-
ModelEntry.from_cls_with_id(LLama3Vision8B),
1816
ModelEntry.from_cls_with_id(PaliGemma2),
1917
*model_entries_from_mult_ids(OpenAIModel),
2018
*model_entries_from_mult_ids(GeminiAPI),

tests/test_llama3.py

Lines changed: 0 additions & 17 deletions
This file was deleted.

0 commit comments

Comments
 (0)