Misc. bug: Vulkan\Llama-server.exe (b7064+) hangs during prompt processing if "--flash-attn on"

[alan-l]

Name and Version

Vulkan\Llama-server.exe on Windows b7064+
Last good version: b7063

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

--flash-attn on

Problem description & steps to reproduce

b7063 with "--flash-attn on" processes prompts and shows results in WebUI.
b7064+ with "--flash-attn on" hangs while processing, needing ctrl+c multiple times to kill program.
b7064+ with "--flash-attn off" proceeds normally.

Vulkan on Intel Iris Xe GPU.

First Bad Commit

Working fine on b7063, versions b7064-b7079 same freezing.

Relevant log output

[alan-l]

Prompts that work: "What's your name?" and "What can you do?" Seems to output correctly.
Example of prompt that hangs: "Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number."

I'm able to roll back to b7063 and run the same prompt with the same command line parameters, and the prompt generates an output. Just tested again with versions from b7064-b7079 and it hangs on prompts like the sample I posted.

[jeffbolznv]

What models? Please post a full command line and log. Does test-backend-ops pass?

[alan-l]

I have no ability to compile and run code, so I can't answer if test-backend-ops pass or not.

Models tested: Gemma 3 4B, Gemma 3 12B, Qwen3 VL 4B, Qwen3 VL 8B, GPT OSS 20B.

Vulkan\Llama-server.exe started with: --threads 4 --ctx-size 32768 --n-gpu-layers 0 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --numa distribute --keep -1 --kv-unified --jinja --host 0.0.0.0 --port 8080 --parallel 4 --model gemma-3-12b-it-qat-UD-Q4_K_XL.gguf --mmproj gemma-3-12b-it-qat-UD-mmproj.F16.gguf

Log's last few lines then freezes. I wait at least 5 mins before killing process:

0.07.245.712 I main: server is listening on http://0.0.0.0:8080 - starting the main loop
0.07.245.717 I srv update_slots: all slots are idle
0.13.077.064 I srv log_server_r: request: GET /props 192.168.1.128 200
0.13.191.523 I srv log_server_r: request: GET /props 192.168.1.128 200
0.14.578.123 I srv log_server_r: request: GET / 192.168.1.128 200
0.15.142.474 I srv log_server_r: request: GET /props 192.168.1.128 200
0.15.457.714 I srv log_server_r: request: GET /props 192.168.1.128 200
0.15.480.849 I srv log_server_r: request: GET /props 192.168.1.128 200
0.15.545.876 I srv update_slots: all slots are idle
0.15.545.913 I srv log_server_r: request: GET /slots 192.168.1.128 200
0.19.827.910 I srv log_server_r: request: GET /props 192.168.1.128 200
0.19.962.764 I srv log_server_r: request: GET /props 192.168.1.128 200
0.20.080.415 I srv params_from_: Chat format: Content-only
0.20.080.909 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.20.082.154 I slot launch_slot_: id 3 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
0.20.082.168 I slot launch_slot_: id 3 | task 1 | processing task
0.20.082.462 I slot update_slots: id 3 | task 1 | new prompt, n_ctx_slot = 32768, n_keep = -1, task.n_tokens = 72
0.20.082.475 I slot update_slots: id 3 | task 1 | n_tokens = 0, memory_seq_rm [0, end)
0.20.082.616 I slot update_slots: id 3 | task 1 | prompt processing progress, n_tokens = 8, batch.n_tokens = 8, progress = 0.111111
0.21.270.921 I slot update_slots: id 3 | task 1 | n_tokens = 8, memory_seq_rm [8, end)
0.21.270.989 I slot update_slots: id 3 | task 1 | prompt processing progress, n_tokens = 72, batch.n_tokens = 64, progress = 1.000000
0.21.271.027 I slot update_slots: id 3 | task 1 | prompt done, n_tokens = 72, batch.n_tokens = 64

[alan-l]

Just finished running my same test on b7083 and it freezes at the same point with --flash-attn on. No freeze after processing prompt if FA is off. Rolled back to b7063 and no freezing with FA on or off, prompt takes seconds to process and output.

[jeffbolznv]

Can you check if you're using the latest driver? I can't think of a good reason why this code wouldn't work on Intel.

[alan-l]

Can you check if you're using the latest driver? I can't think of a good reason why this code wouldn't work on Intel.

I believe they're up to date. My driver version is 32.0.101.6737, and the Intel panel says my Vulkan version is 1.4.309. The Intel panel indicates I'm on the latest version.

Don't know if this additional observation helps or not, but when the processing completes, the GPU load is at 0%, but the cpu load hovers around 10%, which according to task manager is used by the terminal with which I launched llama-server.exe.

Just curious, but would it have anything to do with the change in b7064 with "vulkan: skip all-negative-inf blocks in FA (#17186)"? I'm asking because if changes were done to FA on Vulkan, and I'm having problems when activating FA... It seems that even if I don't set KV quant values but that FA is on, this problem shows up.

[jeffbolznv]

It definitely seems related to that change.

The GPU load being at zero and CPU load being at 10% seems like maybe there was a GPU error or something similar that caused the fence not to be signaled.

[alan-l]

It definitely seems related to that change.

The GPU load being at zero and CPU load being at 10% seems like maybe there was a GPU error or something similar that caused the fence not to be signaled.

I'm GPU-poor and have limited coding skills, but whatever I can do to help resolve this issue, let me know!

[0cc4m]

I do also have an Intel flash attention issue that I haven't had the time to look into yet, it may be related.

Just to be sure this is not another open Intel issue (#17106): @alan-l please try running with environment variables GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 and see if this changes anything. Also try GGML_VK_DISABLE_F16=1.

[alan-l]

I do also have an Intel flash attention issue that I haven't had the time to look into yet, it may be related.

Just to be sure this is not another open Intel issue (#17106): @alan-l please try running with environment variables GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 and see if this changes anything. Also try GGML_VK_DISABLE_F16=1.

@0cc4m I tried as recommended, 8 runs, combination of the 2 env vars requested with and without FA enabled. Same results as I earlier reported.

[0cc4m]

Thank you!

[jeffbolznv]

My driver version is 32.0.101.6737

I think this is like 7 months old, can you try to install a newer one, maybe from https://www.intel.com/content/www/us/en/download/864990/intel-11th-14th-gen-processor-graphics-windows.html?wapkw=iris