Skip to content

Fix SSM dt clamp default for Nemotron-H#1026

Merged
angeloskath merged 1 commit intoml-explore:mainfrom
kernelpool:fix-nemotron-h
Mar 30, 2026
Merged

Fix SSM dt clamp default for Nemotron-H#1026
angeloskath merged 1 commit intoml-explore:mainfrom
kernelpool:fix-nemotron-h

Conversation

@kernelpool
Copy link
Copy Markdown
Contributor

@kernelpool kernelpool commented Mar 20, 2026

Fix that removes incorrect lower bound clamping on SSM time steps for Nemotron-H, which negatively affects output quality. This also matches the HF reference. Results from tests below (4-bit versions):

seq_len=4096, 25 samples, tulu-3-sft-mixture

Model PPL (before) PPL (after) Delta KL div
Nano 4B 4.864 4.845 -0.019 0.006
Nano 30B 3.987 3.946 -0.041 0.034
Super 120B 3.439 3.363 -0.076 0.046

seq_len=4096, 25 samples, calibration_v5.txt

Model PPL (before) PPL (after) Delta
Nano 4B 9.983 9.886 -0.097
Nano 30B 6.673 6.513 -0.159
Super 120B 5.563 5.203 -0.361

@Thump604
Copy link
Copy Markdown

Nice find — this matches the HF reference and the perplexity improvements are significant, especially on the Super 120B.

I run Nemotron-3-Super-120B-A12B (5-bit) daily on M2 Ultra 128GB via the Nemotron-H implementation from PR #992. Will test this fix against my production workload and report back with before/after comparison.

The original time_step_min → time_step_limit mapping was likely cargo-culted from Mamba-1 where it made sense; Nemotron-H's Mamba-2 doesn't need it. +1 for merge.

Copy link
Copy Markdown
Member

@angeloskath angeloskath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks for the fix and the evals.

@angeloskath angeloskath merged commit 6ddfdda into ml-explore:main Mar 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants