In regulated domains such as banking, LLM-based Automatic Speech Recognition (ASR) faces privacy bottlenecks: collecting real speech is costly and legally constrained, making synthetic Text-to-Speech (TTS) an attractive alternative.
However, synthetic speech remains acoustically mismatched with real recordings, and research on this gap has primarily focused on Supervised Fine-Tuning (SFT). We turn to Reinforcement Learning and demonstrate that Group Relative Policy Optimization (GRPO) extracts significantly more from the same synthetic speech than SFT.
The synthetic-only adaptation of the model using GRPO, a critic-free method rewarding low Word Error Rate (WER) hypotheses, reduces WER by 40% relative to SFT (from 36.71% to 22.09%), and a combination of SFT followed by GRPO pushes this further to 45%. We attribute the gain to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment, better anchoring attention to audio, while leaving early-layer representations intact.
When synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.
Blogger's Review: This research highlights the potential of reinforcement learning in the field of speech recognition, especially under data constraints. The GRPO method significantly improves the accuracy of recognizing synthetic speech by enhancing the model's behavioral performance, paving the way for new applications in speech technology.