[Feature]: turboquant: KV cache #296
Description
I'd like to benefit from KV Cache quantization on macOS
https://github.com/OnlyTerp/turboquant
https://github.com/mitkox/vllm-turboquant
https://github.com/scrya-com/rotorquant
{{ message }}
I'd like to benefit from KV Cache quantization on macOS
https://github.com/OnlyTerp/turboquant
https://github.com/mitkox/vllm-turboquant
https://github.com/scrya-com/rotorquant