Recent automatic speech recognition (ASR) systems increasingly incorporate large language models (LLMs) to exploit their semantic knowledge, either via external logit fusion or internal warm initialization. Yet an effective combination of these two strategies remains under‑explored.
This work focuses on warm‑initialized, LoRA‑adapted LLM‑based ASR models where the base LLM is kept intact, and leverages the pre‑adaptation base model for further refinement.
We introduce Hybrid Search, a targeted correction approach motivated by two observations: (1) interaction features between ASR hidden states and base‑LLM hidden states reveal a token’s degree of semantic dependence; (2) selectively refining tokens with high semantic dependence yields far better performance than naïve global LLM‑correction methods such as rescoring or late fusion.
Empirical analysis shows that even after semantic knowledge transfer through warm initialization, the base LLM can still be exploited at inference time to boost accuracy.
Review