Model: Qwen3 Next #16095

pwilkin · 2025-09-18T19:45:57Z

EDIT: README FIRST
This is an implementation of a new type of attention gating in GGML.
Therefore, this implementation will be focused on CORRECTNESS ONLY.
Speed tuning and support for more architectures will come in future PRs.
Please do not spam this threads with reports about performance, especially on backend architectures (CUDA, Vulkan).

CURRENT STATE: rebase complete, ~~waiting for review / decision on chunking~~ something's broken so need to debug

===
It's been a real learning experience, not gonna lie, but if someone with hybrid model implementation experience (@gabe-l-hart ?) has some quick tips, I'd be grateful.

Currently at the stage of "graph builds, but first decode complains about wrong memory model", probably not building the inputs correctly.

Resolves #15940

gabe-l-hart · 2025-09-18T19:55:31Z

I'll try to get into it in more detail soon, but here are a few general thoughts after quickly skimming the PR:

The structure of what you've got smells correct, so it's likely close, but missing something small yet critical
A full repro with the error it's raising would definitely help debug
My debugging process for this would be:
1. Make sure tokenization is solid (print statements as necessary to compare tokens before input)
2. Use llama-eval-callback to dump tensors for a single prefill step
3. Run an identical single prefill with the reference impl (transformers or otherwise), and inject prints as needed to dump tensors along the way
4. Visually comb through them (particularly the sum at each point) to see where things start diverging significantly

bugparty · 2025-09-18T20:47:56Z

It's been a real learning experience, not gonna lie, but if someone with hybrid model implementation experience (@gabe-l-hart ?) has some quick tips, I'd be grateful.

Currently at the stage of "graph builds, but first decode complains about wrong memory model", probably not building the inputs correctly.

Resolves #15940

interesting, maybe we can learn together

ggml/src/ggml.c

ggml/src/ggml-cpu/ggml-cpu.c

pwilkin · 2025-09-19T12:53:24Z

A full repro with the error it's raising would definitely help debug

Running llama-cli -m reference/qwen3_next_500m/Qwen3_Next_500M-8x417M-BF16.gguf -ngl 999 -p "Who are " yields this weird memory error:

#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x000070552b29eb63 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49     ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x000070552b31afdf in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x000070552bb45c31 in ggml_print_backtrace () at /devel/tools/llama.cpp/ggml/src/ggml.c:196
warning: Source file is more recent than executable.
196             waitpid(child_pid, NULL, 0);
#5  0x000070552bb45de5 in ggml_abort (file=0x70552bbcdac8 "/devel/tools/llama.cpp/ggml/src/ggml-backend.cpp", line=189, fmt=0x70552bbcd8af "GGML_ASSERT(%s) failed") at /devel/tools/llama.cpp/ggml/src/ggml.c:230
230             ggml_print_backtrace();
#6  0x000070552bb6091e in ggml_backend_buffer_get_type (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:189
189         GGML_ASSERT(buffer);
#7  0x000070552bb6080e in ggml_backend_buffer_is_host (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:170
170         return ggml_backend_buft_is_host(ggml_backend_buffer_get_type(buffer));
#8  0x000070552c07a114 in llm_graph_input_rs::set_input (this=0x5f11bdf6aea0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:241
241             GGML_ASSERT(ggml_backend_buffer_is_host(s_copy->buffer));
#9  0x000070552c07b03c in llm_graph_input_mem_hybrid::set_input (this=0x5f11bdf6aee0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:437
437         inp_rs->set_input(ubatch);
#10 0x000070552c07b549 in llm_graph_result::set_inputs (this=0x5f11be01ddf0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:480
480             input->set_input(ubatch);
#11 0x000070552c01ddb3 in llama_context::process_ubatch (this=0x5f11c05b5b50, ubatch=..., gtype=LLM_GRAPH_TYPE_DECODER, mctx=0x5f11be00ff00, ret=@0x7fff74d22ea4: 538976288) at /devel/tools/llama.cpp/src/llama-context.cpp:779
779             res->set_inputs(&ubatch);
#12 0x000070552c01f367 in llama_context::decode (this=0x5f11c05b5b50, batch_inp=...) at /devel/tools/llama.cpp/src/llama-context.cpp:1088
1088            const auto * res = process_ubatch(ubatch, LLM_GRAPH_TYPE_DECODER, mctx.get(), status);
#13 0x000070552c025e49 in llama_decode (ctx=0x5f11c05b5b50, batch=...) at /devel/tools/llama.cpp/src/llama-context.cpp:2726
2726        const int ret = ctx->decode(batch);
#14 0x00005f11a2021559 in common_init_from_params (params=...) at /devel/tools/llama.cpp/common/common.cpp:1066
1066                llama_decode(lctx, llama_batch_get_one(tmp.data(), std::min(tmp.size(), (size_t) params.n_batch)));
#15 0x00005f11a1e4a3c0 in main (argc=7, argv=0x7fff74d25968) at /devel/tools/llama.cpp/tools/main/main.cpp:140
140         common_init_result llama_init = common_init_from_params(params);

I'll try to merge the op into the ggml_delta_net function call as @ngxson suggested.

CISC · 2025-09-19T13:28:30Z

A full repro with the error it's raising would definitely help debug

Running llama-cli -m reference/qwen3_next_500m/Qwen3_Next_500M-8x417M-BF16.gguf -ngl 999 -p "Who are " yields this weird memory error:
...
#6  0x000070552bb6091e in ggml_backend_buffer_get_type (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:189
189         GGML_ASSERT(buffer);
#7  0x000070552bb6080e in ggml_backend_buffer_is_host (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:170
170         return ggml_backend_buft_is_host(ggml_backend_buffer_get_type(buffer));
...

The backend buffer is NULL.

ngxson · 2025-09-19T13:40:52Z

#9  0x000070552c07b03c in llm_graph_input_mem_hybrid::set_input (this=0x5f11bdf6aee0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:437
437         inp_rs->set_input(ubatch);

The model doesn't seem to have any recurrence layers. This makes the set input fails due to input node not being present in cgraph.

I'll try to merge the op into the ggml_delta_net function call as @ngxson suggested.

Hmm I think I said the reverse: not to merge it but make the op simple

I feel like this op can be implemented using other ggml ops like mul, mul_mat, sum. Which part of the calculation do you think that can't be constructed using existing ops?

This is the more important question: should we try to implement it using existing ops, or add a new op and spend even more time to optimize it cross all backends?

pwilkin · 2025-09-19T15:10:34Z

Now this is an error I haven't expected to encounter:

GGML_ABORT("not enough space in the context's memory pool");

pwilkin · 2025-09-19T15:18:04Z

The model doesn't seem to have any recurrence layers. This makes the set input fails due to input node not being present in cgraph.

How do I allocate the memory for the linear layers then? I seem to have misunderstood how build_inp_mem_hybrid() works...

yarikdevcom · 2025-09-19T15:34:12Z

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

pwilkin · 2025-09-19T16:43:14Z

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

Added a buymeacoffee link to my profile (do consider first funding the Llama.cpp project itself, though!)

ServeurpersoCom · 2025-09-19T16:55:46Z

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

Added a buymeacoffee link to my profile (do consider first funding the Llama.cpp project itself, though!)

I send a coffee also.

ngxson · 2025-09-20T03:54:48Z

GGML_ABORT("not enough space in the context's memory pool");

Probably there are too many nodes on cgraph, try increasing the limit via llama_context::graph_max_nodes()

src/llama-model.cpp

ngxson · 2025-09-20T05:01:34Z

src/llama-model.cpp

+        Qcur = ggml_reshape_3d(ctx0, ggml_cont(ctx0, Qcur), n_embd_head, hparams.n_head(il), n_tokens);
+        Kcur = ggml_reshape_3d(ctx0, ggml_cont(ctx0, Kcur), n_embd_head, hparams.n_head_kv(il), n_tokens);
+        Vcur = ggml_reshape_3d(ctx0, ggml_cont(ctx0, Vcur), n_embd_head, hparams.n_head_kv(il), n_tokens);


these ggml_cont can be removed if Q/gate are separated. ggml_cont is not recommended when dealing with big tensors

Actually none of these need ggml_cont, Q is 3D already, Q/K are RoPEd so can be views and V can also be a 3D view now.

Edit: sorry, not quite true about V, only if QKV is fused, the weird gate fuse threw me off. Nevertheless, K/V are already contiguous at this point.

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

Are you sure? AFAIK those issues are fixed.

Edit: Also, if there still are issues they will never get fixed if we work around them. :)

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

I think all of these cases are fixed now.

This was an impl of 2D rope that relies on ggml_view: https://github.com/ngxson/ggml-easy/blob/f56e5e499b1f21a4aae73010e9d9582840428457/demo/2d-rope.cpp

It works on CPU and Metal, but doesn't work on CUDA/Vulkan. Couldn't tested on other backends, but feel free to make a PR to address this issue.

Does it still fail? I think these PRs should have addressed the problem:

cuda : fix rope with partial rotation and non-cont src #14580

vulkan: fix rope with partial rotation and non-cont src #14582

yes that seems to work. sorry @pwilkin you will need to manually revert the change where I split Q/gate. the tensor shape for Q will be:

layer.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), { n_embd, n_embd_head_k * n_head * 2 }, 0);

ngxson · 2025-09-20T05:02:45Z

src/llama-model.cpp

+                            layer.ssm_a = create_tensor(tn(LLM_TENSOR_SSM_A, i), { hparams.ssm_dt_rank }, 0);
+                            layer.ssm_beta_alpha = create_tensor(tn(LLM_TENSOR_SSM_BETA_ALPHA, "weight", i), { n_embd, ba_projection_size }, 0);
+                            layer.ssm_norm = create_tensor(tn(LLM_TENSOR_SSM_NORM, "weight", i), { head_v_dim }, 0);
+                            layer.ssm_out = create_tensor(tn(LLM_TENSOR_SSM_OUT, "weight", i), { n_ff, n_embd }, 0);


Shape of LLM_TENSOR_ATTN_Q and LLM_TENSOR_SSM_OUT should not contain n_ff

ngxson · 2025-09-20T05:03:28Z

^ proposed fix for the 3 comments above: 46110e0

src/llama-model.cpp

ggml/src/ggml-cpu/ggml-cpu.c

pwilkin · 2025-09-20T10:26:32Z

@ngxson Thanks, scale_bias was one op I was missing in my endeavors :>

I got an LLM to rewrite the internal delta into tensor logic. After a day of manually fixing that crap, I think I understand it enough to rewrite it myself ;)

ngxson · 2025-09-20T10:31:46Z

Honestly I would prefer taking time to understand the mamba/ssm implementation then writing the code manually. Code written by LLM are mostly attempts for 1-to-1 translation from pytorch --> GGML which looks quite confusing

pwilkin · 2025-09-20T12:14:57Z

Honestly I would prefer taking time to understand the mamba/ssm implementation then writing the code manually. Code written by LLM are mostly attempts for 1-to-1 translation from pytorch --> GGML which looks quite confusing

Yeah, for me getting a rough outline then going over it manually is the best way to learn :)

I tried the "one-to-one" approach and ended up with a graph that wouldn't fit in 16 GB of RAM for a 500M model...

pwilkin · 2025-09-20T21:22:09Z

Aight, I cleaned up the main graph calculation, now I have to figure out how to include conv_states_all in my delta_net function in order to not get the memory error.

M3l-Idk · 2025-11-14T15:12:03Z

Whats going on?

pwilkin · 2025-11-14T16:04:48Z

Whats going on?

Updated the main post. Next stage will be to rebase it on top of current master with the operators added in and look for a solution to the chunking problem.

@ggerganov what would you think about an operation CHUNK that takes a separate graph and performs the graph operations repeatedly on an input tensor? I don't see another way out of the conundrum with the hybrid models which on one way require chunking due to the O(n^3) solve operations and preventing graph overflow on the other hand...

pwilkin · 2025-11-14T18:11:16Z

Lo and behold, I have performed a CLEANUP and this nightmare of a PR is no more, elegant clean version has been uploaded :>

src/llama-model.cpp

src/models/qwen3next.cpp

src/llama-context.cpp

calvin2021y · 2025-11-15T16:55:30Z

with this pr I get:

llama_model_load: error loading model: error loading model hyperparameters: unsupported model architecture
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'lefromage/Qwen__Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf', try reducing --n-gpu-layers if you're running out of VRAM
srv    load_model: failed to load model, 'lefromage/Qwen__Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

lefromage · 2025-11-16T18:46:28Z

this PR is missing break; in llama-model.cpp:2238, at end of case LLM_ARCH_QWEN3NEXT:

diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index 3e49060dd..35b82da41 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -2235,7 +2235,7 @@ void llama_model::load_hparams(llama_model_loader & ml) {
case 80: type = LLM_TYPE_80B_A3B; break;
default: type = LLM_TYPE_UNKNOWN;
}

```
       }
```

       } break;
   default: throw std::runtime_error("unsupported model architecture");

}

but then other error:

load_tensors: loading model tensors, this can take a while... (mmap = false)
llama_model_load: error loading model: missing tensor info mapping for blk.0.ssm_ba.weight
llama_model_load_from_file_impl: failed to load model

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

pwilkin · 2025-11-17T10:39:33Z

I've cleaned up the obvious bugs but there are some regressions, not sure why at the moment, will debug to find out.

pwilkin · 2025-11-17T15:13:03Z

@ggerganov this seems like a bug, and a pretty ugly one at that:

ggml_debug:        g_cum_last_view-0 = (f32)       VIEW(g_cumsum-0 (transposed) (cont){1, 4, 2, 1}, }) = {1, 1, 2, 1}
                                     [
                                      [
                                       [     -6.1265],
                                      ],
                                      [
                                       [    -73.0931],
                                      ],
                                     ]
                                     sum = -79.219696
ggml_debug:             g_cum_last-0 = (f32)       CONT(g_cum_last_view-0{1, 1, 2, 1}, }) = {1, 1, 2, 1}
                                     [
                                      [
                                       [     -6.1265],
                                      ],
                                      [
                                       [    -17.9307],
                                      ],
                                     ]
                                     sum = -24.057196

pwilkin · 2025-11-17T18:47:39Z

Actually, tested on CPU and this is CUDA-specific. @am17an maybe you'd want to take a look? It seems in this specific case CONT is not behaving well.

pwilkin · 2025-11-17T21:59:48Z

Okay, got it. See #17332

pwilkin · 2025-11-17T22:01:57Z

With the CUDA fix the model has been restored to its righteous glory:

> Hello model!
Hello! 😊 How can I help you today? Whether you have a question, need help with a task, or just want to chat—I'm here for you!

> Write me a nice quicksort in Haskell.
Absolutely! Here’s a clean, idiomatic, and *nice* quicksort in Haskell — classic, elegant, and beautifully functional:

quicksort :: Ord a => [a] -> [a]
quicksort [] = []
quicksort (x:xs) =
    quicksort smaller ++ [x] ++ quicksort larger
  where
    smaller = [y | y <- xs, y <= x]
    larger  = [y | y <- xs, y > x]

### Why it’s nice:
- **Concise**: Just 4 lines of logic.
- **Readable**: The intent is crystal clear — partition, recurse, concatenate.
- **Idiomatic**: Uses list comprehensions and pattern matching, which are staples of Haskell.
- **Polymorphic**: Works on any type that supports ordering (`Ord a`).
- **Pure & immutable**: No side effects — pure functional style.

### Example usage:
main :: IO ()
main = do
    print $ quicksort [3, 6, 8, 4, 2, 5, 1]  -- [1,2,3,4,5,6,8]
    print $ quicksort "haskell"              -- "aehklls"

> 💡 *Note: This isn’t the most efficient in practice (due to list concatenation and extra passes), but it’s perfect for learning and showcases Haskell’s beauty.*

Want a faster, tail-recursive, or in-place version? (Even though “in-place” isn’t really a thing in pure Haskell 😉) — just ask!

pwilkin force-pushed the qwen3_next branch from 2f47798 to 178230e Compare September 18, 2025 19:47

github-actions bot added python python script changes ggml changes relating to the ggml tensor library for machine learning labels Sep 18, 2025

ngxson reviewed Sep 19, 2025

View reviewed changes

pwilkin marked this pull request as draft September 19, 2025 08:07

github-actions bot added the examples label Sep 19, 2025

rick-github mentioned this pull request Sep 19, 2025

Qwen3-Next ollama/ollama#12349

Open

ngxson reviewed Sep 20, 2025

View reviewed changes

src/llama-model.cpp Outdated Show resolved Hide resolved

ngxson reviewed Sep 20, 2025

View reviewed changes

src/llama-model.cpp Outdated Show resolved Hide resolved

ngxson reviewed Sep 20, 2025

View reviewed changes

src/llama-model.cpp Outdated Show resolved Hide resolved

ngxson reviewed Sep 20, 2025

View reviewed changes

ggml/src/ggml-cpu/ggml-cpu.c Outdated Show resolved Hide resolved

ngxson mentioned this pull request Sep 20, 2025

mtmd: more optimized build_rope_2d #16126

Closed

This comment has been minimized.

Sign in to view

Qwen3 Next - cleaned up version

ff26271

pwilkin force-pushed the qwen3_next branch from f63e270 to ff26271 Compare November 14, 2025 18:08

Whitespaces and stuff

b87b6d4

pwilkin requested review from CISC and ggerganov November 14, 2025 18:10

DajanaV mentioned this pull request Nov 14, 2025

UPSTREAM PR #16095: Model: Qwen3 Next auroralabs-loci/llama.cpp#210

Open

Correct minor errors

42503dc

github-actions bot added the model Model specific label Nov 14, 2025

CISC reviewed Nov 14, 2025

View reviewed changes

src/llama-model.cpp Outdated Show resolved Hide resolved

src/models/qwen3next.cpp Show resolved Hide resolved

zeerd reviewed Nov 15, 2025

View reviewed changes

src/llama-context.cpp Outdated Show resolved Hide resolved

pwilkin and others added 3 commits November 16, 2025 21:46

Update src/llama-model.cpp

e92d018

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Misc. fixes.

6879383

Clean up code, add missing hybrid qualifier

9d8e103

Did someone transpose the SOLVE_TRI result matrix? Perhaps...

ce554a9

pwilkin added 3 commits November 17, 2025 19:48

Whitespace

12a50b3

Proper tensors for cb calls

f66598e

Fix too relaxed check on CUDA "fast copy" (can_be_transposed) condition

cb187a9

pwilkin mentioned this pull request Nov 17, 2025

Fix too relaxed check on CUDA "fast copy" (can_be_transposed) condition #17332

Open

DajanaV mentioned this pull request Nov 17, 2025

UPSTREAM PR #17332: Fix too relaxed check on CUDA "fast copy" (can_be_transposed) condition auroralabs-loci/llama.cpp#244

Open

Model: Qwen3 Next #16095

Are you sure you want to change the base?

Model: Qwen3 Next #16095

Conversation

pwilkin commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gabe-l-hart commented Sep 18, 2025

Uh oh!

bugparty commented Sep 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwilkin commented Sep 19, 2025

Uh oh!

CISC commented Sep 19, 2025

Uh oh!

ngxson commented Sep 19, 2025

Uh oh!

pwilkin commented Sep 19, 2025

Uh oh!

pwilkin commented Sep 19, 2025

Uh oh!

yarikdevcom commented Sep 19, 2025

Uh oh!

pwilkin commented Sep 19, 2025

Uh oh!

ServeurpersoCom commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngxson commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ngxson Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

ngxson Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

ggerganov Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

ngxson Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

ngxson Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

ngxson commented Sep 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwilkin commented Sep 20, 2025

Uh oh!

ngxson commented Sep 20, 2025

Uh oh!

pwilkin commented Sep 20, 2025

Uh oh!

pwilkin commented Sep 20, 2025

pwilkin commented Sep 18, 2025 •

edited

Loading

ServeurpersoCom commented Sep 19, 2025 •

edited

Loading

ngxson commented Sep 20, 2025 •

edited

Loading

CISC Sep 20, 2025 •

edited

Loading

CISC Sep 20, 2025 •

edited

Loading