This work introduces Vertical Adaptive Layer Skipping (VALSE) to improve inference efficiency of large language models. A theoretical framework is first built, establishing three key results: (i) Theorem 2 provides a closed‑form expected FLOPs expression, linking computational cost of any per‑sample skip schedule to layer‑wise skip probabilities; (ii) Theorems 10 and 11 show that skip‑layer models form a strict subset of the full‑layer function space yet can meaningfully approximate it, accompanied by an explicit separating example; (iii) Proposition 6 reveals a structural duality between VALSE and Mixture‑of‑Experts, positioning vertical depth‑wise sparsity as the orthogonal counterpart of horizontal width‑wise sparsity. Leveraging this theory, a concrete method is proposed: a lightweight difficulty estimator scores each input in the early layers, then per‑layer gates decide whether to skip, allowing arbitrary middle layers to be omitted while deeper layers remain active, thus activating only the necessary depth per sample. Prototype experiments on smaller models demonstrate feasibility.
Review