In the field of speech processing, phone segmentation and recognition are inherently related tasks, yet modern approaches often model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.
Blogger's Review: This study effectively integrates phone segmentation and recognition through phonological activation mapping, showcasing the potential of self-supervised learning in phonetic tasks. The design of lightweight prediction heads reduces computational complexity, enhancing its practicality and flexibility in real-world applications. Notably, its generalization ability to unseen phones indicates promising prospects for future speech processing applications.