Throughputs of Long Sequences #12608

[simmonssong]

Hi, I am testing throughputs of input sequences with different lengths. I found that throughput increases with length on several different models and quantization, is this caused by build-in infrastructure optimization of Llama.cpp?

[onestardao]

interesting catch — yeah some infra optimizations in llama.cpp (like tiled kv-cache) do boost throughput on long sequences,
but there's a weirder side effect: models feel faster while actually getting semantically fuzzier under compression stress.

we ran into this when stress-testing long-form prompts + nested reasoning — throughput went up, but logic fidelity went sideways.

anyway, great question. if you're testing semantic drift under load too, happy to swap notes. some of those failure patterns are... spooky.