Nemotron-Cascade-2: thinking mode infinite loop on Apple Silicon

[roeex5]

Environment

• Model: mlx-community/Nemotron-Cascade-2-30B-A3B-8bit
• mlx-lm version: 0.31.1
• Apple Silicon M1 Max, 64GB unified memory
• Python 3.12

Problem

When running Nemotron-Cascade-2-30B-A3B with enable_thinking=True, the model never generates </think> — it loops indefinitely in the reasoning block and never produces a response.

With enable_thinking=False, generation stops after only a few tokens with poor quality output.

Findings

</think> is token ID 13, which is not included in the default EOS token set ({2, 11}). Without it in the EOS set, the model has no stop condition for the thinking block.

vLLM handles this via --reasoning-parser nemotron_v3, which extends DeepSeekR1ReasoningParser and watches for </think> to transition from reasoning to content. No equivalent exists in mlx-lm for this model.

Workaround attempted

Manually adding token 13 to eos_token_ids stops the thinking phase cleanly. However, the answer phase immediately re-enters thinking, producing incoherent output:

# Phase 1: thinking
tokenizer.eos_token_ids.add(13)
for r in stream_generate(model, tokenizer, prompt=prompt, max_tokens=4096, sampler=sampler):
    thinking += r.text
    if r.finish_reason:
        break
tokenizer.eos_token_ids.discard(13)

# Phase 2: answer — re-enters thinking loop, output is incoherent
prompt2 = prompt + thinking + "</think>\n"
for r in stream_generate(model, tokenizer, prompt=prompt2, ...):
    ...

Comparison

The same model via Ollama (GGUF Q4_K_M) works correctly — thinking terminates naturally and produces coherent output. This suggests the issue is in how mlx-lm handles the thinking state machine for this architecture, not the model weights themselves.

Request

Would it be possible to implement a reasoning parser equivalent for nemotron_h / Nemotron-Cascade in mlx-lm, similar to how vLLM's nemotron_v3 parser handles the <think>/</think> transition?

The model runs well on Apple Silicon (MLX 8-bit loads correctly at ~33GB) and should be significantly faster than the GGUF alternative once thinking mode is handled correctly.

[Thump604]

I run Nemotron-3-Super-120B-A12B on M2 Ultra 128GB daily -- same architecture family. The infinite thinking loop is expected behavior when there's no reasoning parser handling the think/content state machine.

The core problem: Nemotron-H models generate ... as part of their output format, but isn't an EOS token -- it's a state transition signal. The model expects something downstream to detect the transition and switch from reasoning to content mode. Without that, it just keeps generating reasoning tokens forever.

For bare mlx-lm (no server):

The cleanest workaround right now is a stop-word approach. Pass </think> as a stopping string and handle the two phases yourself:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Nemotron-Cascade-2-30B-A3B-8bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 2+2?"}],
    tokenize=False, add_generation_prompt=True
)

# Generate with </think> as stop condition
# The model will emit <think>...reasoning...</think> then stop
response = generate(
    model, tokenizer, prompt=prompt,
    max_tokens=4096,
    # You can check the output for </think> and take everything after it
)
# Split on </think> -- reasoning is before, answer is after
if "</think>" in response:
    reasoning, answer = response.split("</think>", 1)
else:
    answer = response

Your two-phase approach (generate reasoning, then restart generation for answer) breaks because you're creating a new generation context instead of continuing the same one. The model sees the concatenated prompt as a fresh start and re-enters thinking.

For server-based inference (vllm-mlx):

vllm-mlx 0.2.6 has a super_v3 reasoning parser (--reasoning-parser super_v3) that handles this properly. It extends the DeepSeek R1 parser, watches for , and splits the response into reasoning_content and content fields. It also handles enable_thinking=false by suppressing the tag entirely.

Why adding to EOS is wrong:

Adding to EOS would terminate generation entirely when it appears, rather than transitioning to the content phase. The model needs to keep generating after -- that's where the actual answer lives.

A proper solution in mlx-lm's generate API would be stop-word support (generate until a string is seen, then optionally continue). That would cover this use case and DeepSeek R1 and any other thinking model.