Despite the rapid advancements in Automatic Speech Recognition (ASR), handling disfluent speech remains a significant challenge. State-of-the-art systems are often optimized to omit disfluencies, resulting in information loss and hallucinations. Previous work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. To address this gap, we employ a Continual Learning (CL) approach with explicit disfluency tokens.
We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, as well as a consistent cross-attention head mechanism shared across CL methods.
Blogger's Review: This study effectively tackles the issue of information loss in disfluent speech recognition by providing an innovative continual learning framework. By incorporating disfluency markers, the model not only enhances recognition accuracy but also maintains sensitivity to general knowledge, showcasing the potential of continual learning in real-world applications.