In post-training alignment, the diversity of large language models (LLMs) often decreases, leading to a phenomenon known as mode collapse. Unlike previous studies attributing this effect to algorithmic limitations, we identify a fundamental data-level driver: typicality bias in preference data, where annotators systematically favor familiar text due to established cognitive psychology findings.
We theoretically formalize this bias, empirically verify it on preference datasets, and demonstrate its central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling (VS), a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities").
Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe that more capable models benefit more from VS. In summary, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.
Blogger's Review: Verbalized Sampling presents an innovative prompting strategy that effectively enhances the creative output of LLMs while maintaining accuracy and safety. This method not only addresses mode collapse but also opens new avenues for diverse applications of LLMs. The combination of theoretical and empirical research provides valuable insights for future model optimization.