Language models are increasingly employed to generate synthetic survey samples, also known as silicon samples. Decision models form a new class of language models that output a probability for each point on a rating scale instead of textual responses. This study asks whether such models, especially the newest generative LLMs, can replace human respondents in a psychometric survey.
We evaluated fourteen decision models from eleven providers and forty‑seven generative LLMs on an open Lithuanian organisational‑psychology questionnaire (n=263, 68 items). The assessment used 100 real respondent profiles, the Psychometric Similarity Score (PSS), and statistical baselines.
The best generative model, MiniMax M3, achieved a PSS of 0.813, comparable to a Gaussian copula fit based only on other respondents (0.824) and below an in‑sample copula fit (0.927). A second set of 100 real people scored lower than the best models, indicating that at this sample size PSS cannot reliably rank models against real respondents.
Across ten product lines, five newest releases scored below an earlier release of the same line beyond resampling noise, and none showed improvement. With the default rounded output, thirteen of the fourteen decision models fell below the generative reference gpt-5.4-mini.
Among the forty‑seven generative models, thirty‑three reproduced the pattern of item correlations better than any decision model. Averaged over respondents, the probability vectors align much more closely with human item marginals than rounded answers, and they rank respondents about as well as a profile‑regression fit on 163 real participants.
However, when forecasting each individual’s own answers, every decision model performed worse than the answer frequencies of other respondents, mainly because its average vector differs from theirs. Profile regression improves these frequencies by only 1.0%, suggesting that profile fields convey little about how a person answers. Synthetic respondents built on such profiles should therefore aim for sample‑level fidelity.
Review