Non-record: BitNet Ternary — 65M params in 15.9MB (1.1932 BPB)

[chrislovescoding]

Summary

Ternary weight quantization ({-1, 0, +1} at ~1.58 bits/weight) enables fitting 65M parameters in a 15.9MB artifact — 3x the parameter count of standard int6 submissions (~22M params) at similar artifact size.

This explores a fundamentally different optimization axis: instead of aggressively quantizing a small model, we train a much larger model with extreme quantization from the start. No other submission in this competition has attempted ternary/BitNet-style training.

Key Results

Metric	Value
Model params	64,529,040
Artifact size	15,878,267 bytes
Post-quant val_bpb	1.2271
Sliding window val_bpb (stride=64)	1.1932
Quantization gap	0.0003 BPB (near-zero)
Steps	5,026 in 600s on 8xH100

Approach

• Architecture: 12 layers, 768 dim, 12/6 GQA heads, 3x MLP (hidden=2304), LeakyReLU(0.5)-squared, U-Net skip connections
• Ternary STE: Full-precision weights maintained by Muon optimizer; forward pass quantizes to {-1, 0, +1} via Straight-Through Estimator with per-row mean-absolute scaling
• Activation schedule: Full-precision for first 30% of wallclock, ternary STE for remaining 70%
• Compression: Ternary int8 values + zlib-9 (zstd-22 would save ~1MB further)
• Eval: Sliding window stride=64

Key Findings

1. Ternary training works at 65M scale in 10 minutes — loss fully recovers after the ternary transition (spike from 1.30 to 1.36 BPB, then recovery to 1.23)
2. Near-zero quantization gap (0.0003 BPB) because the model is trained with ternary STE
3. 3x more parameters fit in the same budget vs int6 — opens a new frontier for parameter-constrained LMs
4. The ternary approach is orthogonal to and composable with other competition techniques (GPTQ, XSA, EMA, etc.)

Test Plan

• Runs in <10 minutes on 8xH100 SXM
• Artifact under 16MB (15.9MB)
• Reproducible (single seed, deterministic)
• Sliding window eval completes within eval budget