Modern neural networks exhibit predictable scaling trends, yet the underlying mechanisms remain obscure. Traditionally, redundancy is assessed via component importance or representational similarity, both serving as indirect proxies. We treat redundancy as an input‑conditioned dynamic relation: intermediate computational states are functionally redundant when they elicit similar downstream responses.
To characterize this functional substitution directly, we introduce Conditional Functional Substitutability (CFS). CFS uncovers functional relations and reduction opportunities that conventional importance‑ or similarity‑based metrics miss.
Experiments across modalities and Transformer families reveal systematic functional reorganization as scale increases. Controlled scaling studies show that performance gains do not necessarily track growth in substitutability; fixed‑capacity models with more independent functional structure perform better, offering a functional explanation for diminishing returns.
Predicted CFS enables dynamic computation with a superior performance‑computation trade‑off compared to importance‑based component selection, suggesting new avenues for redundancy‑aware computation and more efficient model scaling.
Review