Skip to content

docs: add HIP/AMD NaN warning for q8_0/turbo3 on large K-norm models #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brosequist wants to merge 1 commit into
base: main
Choose a base branch
from
+4 −0
Open

docs: add HIP/AMD NaN warning for q8_0/turbo3 on large K-norm models #66

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/turboquant-recommendations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ These configurations showed promising results but have less validation depth:
| Q8_0 weights | `-ctk q8_0 -ctv turbo2` | phi-4 +3.1% |
| Q4_K_M, Qwen2.5-7B (AMD) | `-ctk q8_0 -ctv turbo3` | NaN on HIP (Metal gets +2.0%). HIP-specific, under investigation |

> ⚠️ **WARNING: q8_0/turbo3 produces NaN on HIP/AMD with models that have large K norms**
> (e.g. Qwen2.5-7B where K norms can reach 274). This is under active investigation.
> **Safe AMD alternative: q8_0/turbo4.**

### Boundary V (auto-enabled for turbo2-V)

A layer-aware V compression strategy that protects the first 2 + last 2 layers with q8_0-V while compressing all remaining layers with turbo2-V. **Auto-enabled when `-ctv turbo2` is set** on recent builds. Opt-out: `TURBO_LAYER_ADAPTIVE=0`. On older builds, activate with `TURBO_LAYER_ADAPTIVE=7`.
Expand Down