Abstract
Automatic speech recognition (ASR) is currently dominated by autoregressive decoders that emit one token at a time. This paper investigates whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text through uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models.
A frozen Whisper encoder provides acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters allow the frozen backbone to attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock.
The resulting model achieves a 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Blogger's Review: This paper presents an innovative audio transcription method leveraging the advantages of discrete diffusion models, significantly enhancing the parallel processing capability of speech recognition. The approach not only reduces word error rates but also demonstrates adaptability in multilingual environments, showcasing its important application potential and research value.