We scale the conventional sub‑150M pre‑training recipe from 53.5 M to 109.7 M parameters while keeping the method fixed (Qwen‑3‑style decoder with grouped‑query attention, RoPE, SwiGLU, RMSNorm, QK‑Norm and a z‑loss; training data FineWeb‑Edu) and only changing the geometry to a deep‑and‑thin 23‑layer × 576‑hidden design. The larger model improves across all benchmarks: BLiMP rises from 78.1 to 81.3, ARC‑Easy from 51.4 to 52.5, WikiText‑2 byte‑perplexity drops from 2.04 to 1.95. Its 81.3 % BLiMP score essentially matches GPT‑X2‑125M (81.28) with about 12 % fewer parameters. Notably the 110 M model uses fewer training tokens (≈ 8 B vs 12 B), so the gain comes from capacity and depth rather than more data. Both models are deliberately conventional; this report serves as a clean scaling control and the baseline rung (R0) for an ablation study.
The ablation ladder proceeds as follows: value residuals (R1) and the Muon optimizer (R2) lift ARC‑Easy by a cumulative +3.6 points (52.5 → 56.1) while BLiMP stays flat, so they are kept. A diverse data blend (R3) and two logit‑distillation settings (R4a/R4b) are not kept – honest negatives. R3 pins ARC‑Easy to FineWeb‑Edu’s educational filtering rather than raw diversity; distillation from a 1.7 B teacher can reach class‑leading ARC‑Easy (56.99, matching GPT‑X2‑125M) but incurs a perplexity penalty, and dialing KD down erases the benefit – thus R2 remains the best kept stack.
Review