We train the 20‑billion‑parameter MoE model GPT‑OSS‑20B under the MXFP4 microscaling format using the open‑source ALTO recipe. The recipe combines hybrid 1D/2D block quantization, randomized Hadamard transforms (RHT) and stochastic rounding (SR) from Abecassis et al., with a simplified weight de‑oscillation scheme adapted from TetraJet‑v2.
On the C4 validation set with a global batch size of 16, MXFP4 + RHT + SR + de‑oscillation reaches a loss of 3.3350 after 16 128 steps, only 0.007 above the BF16 baseline (3.3283). This closes roughly half of the gap left by MXFP4 without de‑oscillation (3.3418). Measured by steps to hit the MLPerf quality target (loss 3.34), de‑oscillation reduces the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%.
Negative end‑to‑end results were observed for differential gradient estimation, outlier clipping and macro‑block scaling, despite their operator‑level SNR improvements. All experiments use simulated MXFP4 kernels on AMD MI300 hardware, and wall‑clock speed has not been optimized.
Review