Model is loaded twice with iGPU Shared Memory

[jhemmond]

What is the issue?

Running the pre-compiled Windows version found here: #7 (comment)

My AMD 7735HS with 680m iGPU is running my models at 30-50% faster rate than on CPU alone. Allocating 16GB for Graphics from the BIOS, the entire model layers are loaded in the iGPU shared part of the memory, but I noticed that normal system RAM is also getting filled with exactly the same amount.

Important to note this is not breaking the functionality, but in a 32GB system, it's making the remaining 16 GB of RAM unusable anytime I load a 14B model with 4k context window. The same does not occur with LM studio when using the Vulkan Backend despite it using Llama.cpp inference engine too.

OS

Windows

GPU

AMD

CPU

AMD

Ollama version

No response

[jhemmond]

I was able to address this mainly by disabling mmap() when loading the model into vRAM.

[McBane87]

I don't see any difference with --no-mmap option. This Vulkan fork still double allocates the same data (?) in VRAM and RAM. Also on Linux.

[jhemmond]

Interesting. I didn't play around with Linux yet. Windows still works fine with the latest file you compiled today. I can only reproduce the issue when I disable --no-mmap. I keep --no-mmap disabled to avoid this.

Windows 11 24H2
Ollama 0.6.0-2d443b3-vulkan
AMD 7735HS with 680m iGPU

[McBane87]

Hmm...OK weired. So are you sure your normal RAM is not getting allocated the same amount as your VRAM, when loading a model? Because if I look into my Task Manager and load a model, I can see that memory gets allocated on both. And it's approx the same amount.

[jhemmond]

Positive! When I turn off the -no-mmap, the behavior you described happens exactly, and disappears once I turn off the feature.

Important to note. I'm testing with models that fit entirely inside my iGPU allocated memory. I use Open WebUi to pass the parameters.

[McBane87]

By saying you "turn off --no-mmap", you meant that you set --no-mmap option? So basically you enable the option --no-mmap. Correct?

But still weired. I'm using a 7900XTX with 24GB VRAM. So the models I'm trying definitley fit completely in my VRAM. Still this double allocation.

Can you maybe share the logoutput of your ollama, when you load the model with this option? Specifically I'm looking for this line:

level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\mcbane\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model G:\\LLMs\\Ollama\\blobs\\sha256-4580f0c073733bb7d3b608ff21b852c7a0fd721297cab27c52f134eaf76978f8 --ctx-size 2048 --batch-size 512 --n-gpu-layers 35 --threads 12 --parallel 1 --port 62617"

[jhemmond]

The screenshot of my Open WebUI custom parameters - https://imgur.com/a/we65NMv
Also, Qwen 2.5: 14B fully loaded in 16GB iGPU taking 13GB of memory. You can see my Windows 11 and a few other applications running alongside Ollama and Open WebUI is all loaded in roughly 12GB of System RAM. For context, i have a 48GB total RAM, with 16 GB allocated to iGPU. - https://imgur.com/mpqO87V

Here's pretty much the full context for your request:

time=2025-03-15T13:41:17.276-04:00 level=WARN source=amd_windows.go:172 msg="amdgpu detected, but no compatible rocm library found. Please install ROCm"
time=2025-03-15T13:41:17.276-04:00 level=WARN source=amd_windows.go:56 msg="unable to verify rocm library: no suitable rocm found, falling back to CPU"
time=2025-03-15T13:41:17.285-04:00 level=INFO source=types.go:137 msg="inference compute" id=0 library=vulkan variant="" compute=1.3 driver=1.3 name="AMD Radeon(TM) Graphics" total="16.0 GiB" available="15.2 GiB"
time=2025-03-15T13:41:19.489-04:00 level=WARN source=types.go:514 msg="invalid option provided" option=keep_alive
time=2025-03-15T13:41:19.555-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.key_length default=128
time=2025-03-15T13:41:19.555-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.value_length default=128
time=2025-03-15T13:41:19.556-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.key_length default=128
time=2025-03-15T13:41:19.556-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.value_length default=128
time=2025-03-15T13:41:19.556-04:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\PC.ollama\models\blobs\sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 gpu=0 parallel=1 available=15504831546 required="10.3 GiB"
time=2025-03-15T13:41:19.591-04:00 level=INFO source=server.go:105 msg="system memory" total="31.7 GiB" free="21.1 GiB" free_swap="19.7 GiB"
time=2025-03-15T13:41:19.591-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.key_length default=128
time=2025-03-15T13:41:19.591-04:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.value_length default=128
time=2025-03-15T13:41:19.591-04:00 level=INFO source=server.go:138 msg=offload library=vulkan layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[14.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="10.3 GiB" memory.required.partial="10.3 GiB" memory.required.kv="1.5 GiB" memory.required.allocations="[10.3 GiB]" memory.weights.total="8.9 GiB" memory.weights.repeating="8.3 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="676.0 MiB" memory.graph.partial="916.1 MiB"
llama_model_loader: loaded meta data with 34 key-value pairs and 579 tensors from C:\Users\PC.ollama\models\blobs\sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen2.5 14B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = Qwen2.5
llama_model_loader: - kv 5: general.size_label str = 14B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-1...
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 14B
llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-14B
llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 14: qwen2.block_count u32 = 48
llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
llama_model_loader: - kv 16: qwen2.embedding_length u32 = 5120
llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 13824
llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 40
llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 8
llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 22: general.file_type u32 = 15
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 33: general.quantization_version u32 = 2
llama_model_loader: - type f32: 241 tensors
llama_model_loader: - type q4_K: 289 tensors
llama_model_loader: - type q6_K: 49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 8.37 GiB (4.87 BPW)
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 1
print_info: model type = ?B
print_info: model params = 14.77 B
print_info: general.name = Qwen2.5 14B Instruct
print_info: vocab type = BPE
print_info: n_vocab = 152064
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-03-15T13:41:19.795-04:00 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\Users\PC\AppData\Local\Programs\Ollama\ollama.exe runner --model C:\Users\PC\.ollama\models\blobs\sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 8 --no-mmap --parallel 1 --port 56994"
time=2025-03-15T13:41:19.798-04:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-15T13:41:19.798-04:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding"
time=2025-03-15T13:41:19.799-04:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error"
time=2025-03-15T13:41:19.824-04:00 level=INFO source=runner.go:931 msg="starting go runner"
load_backend: loaded CPU backend from C:\Users\PC\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | matrix cores: none
load_backend: loaded Vulkan backend from C:\Users\PC\AppData\Local\Programs\Ollama\lib\ollama\vulkan\ggml-vulkan.dll
time=2025-03-15T13:41:19.890-04:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 compiler=cgo(clang)
time=2025-03-15T13:41:19.891-04:00 level=INFO source=runner.go:991 msg="Server listening on 127.0.0.1:56994"
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon(TM) Graphics) - 16128 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 579 tensors from C:\Users\PC.ollama\models\blobs\sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen2.5 14B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = Qwen2.5
llama_model_loader: - kv 5: general.size_label str = 14B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-1...
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 14B
llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-14B
llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 14: qwen2.block_count u32 = 48
llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
llama_model_loader: - kv 16: qwen2.embedding_length u32 = 5120
llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 13824
llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 40
llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 8
llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 22: general.file_type u32 = 15
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 33: general.quantization_version u32 = 2
llama_model_loader: - type f32: 241 tensors
llama_model_loader: - type q4_K: 289 tensors
llama_model_loader: - type q6_K: 49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 8.37 GiB (4.87 BPW)
load: special tokens cache size = 22
time=2025-03-15T13:41:20.050-04:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model"
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 0
print_info: n_ctx_train = 32768
print_info: n_embd = 5120
print_info: n_layer = 48
print_info: n_head = 40
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 5
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: n_ff = 13824
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 32768
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 14B
print_info: model params = 14.77 B
print_info: general.name = Qwen2.5 14B Instruct
print_info: vocab type = BPE
print_info: n_vocab = 152064
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors: Vulkan0 model buffer size = 8148.38 MiB
load_tensors: CPU model buffer size = 417.66 MiB

[jhemmond]

One correction. The Gemma 3 model is giving me this issue despite having -no-mmap disabled. The rest of the models work as expected.

[McBane87]

One correction. The Gemma 3 model is giving me this issue despite having -no-mmap disabled. The rest of the models work as expected.

Thanks for this info. Actually I only tested this so far with gemma3. In this case it makes sense it didn't made any difference. Still weired that gemma3 behaves different. Maybe the newly implemented support is ignoring the option by accident.

[McBane87]

I created a patch 04-disable-mmap-vulkan.patch (see here or here), which will make the --no-mmap applied by default. Newest windows build, I provided, has this already applied. But still, it not fixes this for gemma3 only for other models.

[jhemmond]

Thanks for this. Someone confirmed that Ollama has a memory estimation issue and submitted a PR for it. Let’s hope they fix it soon.

…

On Sun, Mar 16, 2025 at 4:27 AM, McBane87 ***@***.***(mailto:On Sun, Mar 16, 2025 at 4:27 AM, McBane87 <<a href=)> wrote: I created a patch 04-disable-mmap-vulkan.patch (see [here](#7 (comment)) or [here](#7 (comment))), which will make the --no-mmap applied by default. Newest windows build, I provided, has this already applied. But still, it not fixes this for gemma3 only for other models. — Reply to this email directly, [view it on GitHub](#18 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AYIQWA4N4BU7J2LKRHIRS4T2UUYVJAVCNFSM6AAAAABYDD64QKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRXGI3DGNRVGU). You are receiving this because you authored the thread.Message ID: ***@***.***> [McBane87] McBane87 left a comment [(whyvl/ollama-vulkan#18)](#18 (comment)) I created a patch 04-disable-mmap-vulkan.patch (see [here](#7 (comment)) or [here](#7 (comment))), which will make the --no-mmap applied by default. Newest windows build, I provided, has this already applied. But still, it not fixes this for gemma3 only for other models. — Reply to this email directly, [view it on GitHub](#18 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AYIQWA4N4BU7J2LKRHIRS4T2UUYVJAVCNFSM6AAAAABYDD64QKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRXGI3DGNRVGU). You are receiving this because you authored the thread.Message ID: ***@***.***>