Model improvements: FSDP dtype + MoE dispatch + ACT deadlock fix + stochastic-depth (Option B)#56

Open

pzerfos wants to merge 4 commits intokyegomez:mainfrom

pzerfos:fix/fsdp-mixed-precision-dtype

Commits on Apr 28, 2026

fix(model): FSDP mixed precision dtype compatibility
Petros Zerfos
committed
feat(model): MoE grouped dispatch + ACT FSDP deadlock fix + code review fixes + tests

Petros Zerfos
and
claude
committed
feat: stochastic-depth training (Option B)

Petros Zerfos
and
claude
committed
feat(training): add 1B FineWeb-Edu training script with stochastic-depth toggle

Petros Zerfos
and
claude
committed