Layer dropout (also known as stochastic depth) has been shown to speed up training, improve accuracy, and increase robustness to zero‑shot layer pruning in both language and vision Transformers. As models and datasets grew, layer dropout largely vanished from large language model (LLM) pre‑training recipes. Prior work reported that dropout might hurt accuracy, yet no systematic quantification or mitigation was provided. This work conducts a comprehensive study of layer dropout in state‑of‑the‑art LLM training, offering best‑practice guidelines and scaling analysis. All experiments ran on Cerebras CS‑3, covering models from 271 M to 8.2 B parameters, up to 160 B tokens, and more than 2 400 training runs.
Key findings: with optimal layer distribution, time schedule, and optimizer hyper‑parameters, layer dropout reduces training loss at the same FLOPs budget; for a fixed number of training steps, it achieves equal or lower validation loss while saving up to 25 % of training FLOPs. Post‑training, layer dropout enables techniques such as early exit, intermediate‑layer skipping, and self‑speculative decoding, delivering up to 1.5× inference speed‑up with negligible accuracy loss.
In summary, layer dropout should be reinstated in LLM pre‑training pipelines, providing substantial efficiency gains during both training and inference.
Review