Depth‑recurrent language models repeatedly apply a small stack of layers per token, decoupling per‑token compute from the total parameter count. This design allows the model to adjust its effective depth at inference without adding new parameters.
The common way to gauge whether such a model truly exploits its depth is to truncate the depth at test time, keep only a fraction of the layers, plot quality versus retained fraction, and read off the slope. The method is cheap and requires no retraining, but it hides a serious flaw: a single truncation changes three aspects at once – the number of block applications, the amount of distinct computation performed, and whether the read‑out head sits on an out‑of‑distribution residual stream. The observed slope therefore conflates all three factors, yet it is usually interpreted as reflecting only the second.
To disentangle these factors we introduce the Depth Control Protocol (DCP). DCP consists of three positive controls that each isolate one factor while varying the others, a negative control that applies the same interventions to dense transformers to rule out protocol artifacts, and a controlled training intervention to verify causality.
The pivotal control runs the full budget of block applications while performing only a single distinct iteration. This setup is realizable only in architectures with depth‑wise weight sharing, because in a dense network repeating a layer yields a completely different model rather than the same model in an alternative configuration.
With DCP we can separately measure the impact of block count, computational diversity, and residual‑stream distribution on performance, providing a more trustworthy assessment of depth utilization.
Review