In this work we systematically examine the probability distributions produced by large language models (LLMs) for three choice domains: air travel, restaurant selection, and consumer products. Our experiments span nine open‑weight models and evaluate sampling under various temperatures, greedy decoding, and perturbations of prompts and item ordering. All models exhibit self‑coherence: the most probable outcomes stabilize quickly across repeated draws. At the same time, substantial discordance appears across model families and scales, with even the top‑ranked outcomes lacking consensus. This divergence persists across all three domains and remains robust to temperature changes, decoding strategies, and prompt tweaks. The findings indicate that the choice of model influences the generated preference distribution far more than the wording of the prompt, challenging the common assumption that sufficiently capable LLMs can replace survey respondents and yield similar distributions.
Review