Decoder hidden states across layers of large language models often encode complementary information, yet speculative decoding typically relies on a few layers chosen by fixed position heuristics.
Selecting an optimal subset of layers is a combinatorial problem that becomes infeasible to exhaustively evaluate even for moderately deep models.
We introduce a principled multi‑layer selection framework based on two complementary criteria: informativeness measured by matrix entropy and diversity quantified by centered kernel alignment (CKA).
We analyze layer‑wise Shannon entropy, aggregate entropy under hidden‑state concatenation, and pairwise CKA dynamics across depth, revealing distinct informational and geometric compartments in decoder representations.
Guided by these observations, we propose entropy‑CKA selection algorithms that favor sets that are highly informative yet minimally redundant. We also add token‑entropy filtering to discard hidden states linked to low‑predictive‑entropy tokens, improving the reliability of selection statistics.
In EAGLE3‑style speculative decoding, the selected sets outperform fixed baselines, including the default $[2, N//2, N-3]$ rule, and clarify why some defaults remain competitive.
Our results offer a practical recipe for multi‑layer hidden‑state selection, insights into the geometry and information of LLM decoder states, and a foundation for extending the analysis to related applications.
Review: By marrying information‑theoretic metrics with representation geometry, the study delivers an interpretable and reproducible approach to layer selection for speculative decoding, meriting further testing on larger models.