Development Roadmap (2025 Q4)

[ErvinXie]

KTransformers Roadmap - 2025 Q4

Focus

• Usability: Easy installation on x86 + NV GPU, more documents and FQA.
• Model Coverage: K2-Thinking INT4 native support, Qwen3 Finetune.

Usability

• Refactoring KTransformers structure. Refactor: restructure repository to focus on kt-kernel and KT-SFT modulesq recon #1581 @SkqLiao
• Docker start of sglang-kt inference. @SkqLiao
• Local quantization scripts. @ouqingliang
• Documents: installation guide, coverage infomation, contribution guide and FAQ. @SkqLiao @ErvinXie

Model Coverage

• K2 Thinking INT4 native support [Feature] Support Native K2 Thinking #1598 . @ouqingliang @chenht2022
• Support Qwen3 series finetune. @JimmyPeilinLi and community contributors. [Feature] KTransformers Fine-tuning Feature Compatibility & Key Enhancements Support Tracking #1575 add qwen3 attn #1602
• Qwen3 VL inference support, TDB.

Performance / Features

• Layerwise prefill. @chenht2022
• AVX2 Support for inference and finetune. @SkqLiao @JimmyPeilinLi
• AMD Adaption @KMSorSMS [Feature] Support amd blis optimizatioin #1601
• EPLB for inference @chenht2022

Contribution / Maintenance

• Build CI workflow for kt-kernel. TBD
• Merge KT-SFT and kt-kernel @JimmyPeilinLi
• Bi-weekly Office Hour @KMSorSMS

---

Any contribution is welcomed, please email ervinxie@qq.com if you want to join development WeChat group.

[magikRUKKOLA]

Guys, the first thing you should do is to throw away the flashinfer.

Why would you aim to optimize the framework for the Intel CPU etc. while you clearly can't reliably process a long context? What's the point?

[ErvinXie]

We already integrated KTransformers into sglang, so the GPU part is maintained by sglang. CPU/GPU arch is able to reliably process long context. The prefill speed will increase as we release more features like layer-wise prefill. Could you tell me why we should throw away flashinfer?
Thanks for your advice. @magikRUKKOLA.

[magikRUKKOLA]

@ErvinXie

Here is a simple experiment you can try-out.

ikawrakow/ik_llama.cpp#936 (comment)

prefill the MoE LLM (671b+) running via ktransformers (flashinfer) by the query above (80k ctx+). Repeat ten times (with dropping of the prefill cache). I am pretty sure you will get gibberish in 30% of the cases.

[KMSorSMS]

@ErvinXie

Here is a simple experiment you can try-out.

ikawrakow/ik_llama.cpp#936 (comment)

prefill the MoE LLM (671b+) running via ktransformers (flashinfer) by the query above (80k ctx+). Repeat ten times (with dropping of the prefill cache). I am pretty sure you will get gibberish in 30% of the cases.

Didn't catch your point. Did you mean the flashinfer backend has bugs in our ktransformer backend? Or just the flashinfer's own has bugs for the long context like 80k above? If our usage of the flashinfer backend has bugs, in our new roadmap, we have migrated the GPU side and server part to the sglang project, which may choose the flashinfer backend.

[magikRUKKOLA]

@KMSorSMS

So basically you threw away the flashinfer by delegating the task to the SGL. So now its possible to use FA3, flashmla etc. via SGL in ktransformers? Its already working for MoE 671b+ CPU+GPU hybrid inference for the 80k ctx+? Okay, what about the --seed parameter? Was it ever implemented?

[KMSorSMS]

So basically you threw away the flashinfer by delegating the task to the SGL. So now its possible to use FA3, flashmla etc. via SGL in ktransformers? Its already working for MoE 671b+ CPU+GPU hybrid inference for the 80k ctx+?

Yes. We have tested 80k+ ctx without gibberish.

Okay, what about the --seed parameter? Was it ever implemented?

As long as sglang supports the seed parameter, we also support it. Since we just modified the routed expert moe part of sglang to use our backend, and you can check the usage of kt-kernel + sglang here:https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2-Thinking.md. Also, there is an official blog from sglang:https://lmsys.org/blog/2025-10-22-KTransformers/