Abstract
Speech-to-text alignment refers to finding the temporal boundaries of each word in the audio. Some models provide this alignment directly, while others do not. Connectionist Temporal Classification (CTC) and transducer models have alignment by construction, whereas attention-based encoder-decoder (AED) models and speech large language models (LLMs) typically derive word timings from attention weights. All these signals reside on the encoder frame grid, which limits their temporal precision.
We investigate a generic gradient-based alignment applicable to any differentiable ASR model. We compute the gradient of each teacher-forced token log probability with respect to the input, reduce it to a per-frame saliency, and decode the resulting matrix into word boundaries using a single dynamic programming pass. This method requires no training, no model modification, and no alignment heads, and functions across all model families including speech LLMs, aligning on the input grid rather than the coarser encoder grid.
We evaluate it on sixteen models from four families, using read (TIMIT) and spontaneous (Buckeye) speech, comparing each against the model's own native or attention-based alignment. Our findings indicate that the gradient yields usable alignment for every model, often trailing a strong native aligner but outperforming it where the native alignment is weak, such as in streaming models. Its main drawback is the computational cost of one backward pass per token.
Blogger's Review: The gradient-based alignment method proposed in this paper demonstrates versatility and flexibility, effectively adapting to various ASR models without complex modifications. This innovation not only enhances alignment precision but also opens new avenues for future speech recognition systems. Its advantages in streaming models are particularly noteworthy, potentially driving advancements in real-time speech recognition technology.