No Difference in tokens/sec - Ministral3 8B Q5_K_M

[MrMuhannadObeidat]

I used the repo to rebuild llama-cpp from scratch to a different dest compared to original llama-cpp. I am comparing performance of the same base model being executed with same command line parameters using llama-server -m for turbo3 and turbo4. Not seeing any improvement in tokens/second before and after. Actually before the speed of generation is better than after. I am using MAC M1 with 32GB RAM.

[MrMuhannadObeidat]

I missed the part where you highlight the fact that tokens/sec may actually degrade with the added compression of KV cache. I tried with turbo3, do not see noticeable degradation but certainly see major impact on memory consumption and the ability to use much bigger context windows.
Question I have now, is can you take advantage of this from within lm studio or ollama?

[zrlhk]

这个只是对kv缓存压缩，所以只是提升了最大推理上下文的大小。对模型量化压缩和推理速度，是没有提升的。
原来10G显存，如果是一个9b模型，最大上下文128k可能就OOM了。现在压缩后，就可以支持128k上下文了。

[zekrom-vale]

You can just use it like Ollama or LM studio without the slow development or wrapper overheads. I use llama cpp directly and use it with router mode and many llms configured with models.ini and interface it with Open Web UI you can find others too. It's nice, but it has a lot more overhead in setting it up.

Use llama-server --host 0.0.0.0 --port 8080 --models-preset /path/to/ini/models.ini and access it through an Open AI compatible UI like Open Web UI at http://<your-ip>:8080. Or 127.0.0.1 as the loop-back if not connecting to a different computer as I do.

Here is an example model ini file I use for my 5070ti.
models.ini.txt