Large language models (LLMs) are increasingly used to simulate diverse human opinions in open-ended tasks such as synthetic surveys, focus group modeling, and public opinion prediction. However, LLM outputs exhibit systematic opinion homogenization. Practitioners have explored various interventions to increase diversity, but the evaluation landscape remains fragmented, making it difficult to attribute gains to specific components.
To advance a more scientific understanding of LLM output diversity, we designed a factorial experiment that separates two primary intervention dimensions: input conditioning (operationalized through persona depth) and interaction architecture. We evaluated all conditions on 100 real-user open-ended questions across 7 models, measuring diversity with multiple complementary metrics.
Our findings challenge several common assumptions. First, more persona detail does not monotonically increase diversity. The initial step of persona conditioning captures the majority of the gain, while further elaboration with demographic detail does not consistently improve and can reduce diversity on some models.
Second, rather than seeking a single best interaction architecture, we find that different architectures explore largely non-overlapping opinion regions. Combining multiple architectures yields broader coverage than optimizing any one.
Third, commonly attempted low-cost alternatives such as raising sampling temperature and adding diversity instructions produce negligible effects compared to structured interventions. Overall, our work demonstrates that diversity is not a product of scaling along any single dimension, but is highly sensitive to the structural form and combination of interventions.
Blogger's Review: This research profoundly reveals the complexity of achieving opinion diversity in LLMs, emphasizing the need to focus on the structural and combinatorial effects of interventions rather than merely increasing quantity. A scientific understanding of diversity will drive more effective model applications.