Abstract
Silicon sampling, which uses large language models (LLMs) to simulate human survey respondents, has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, risking conflation of pattern matching with coherent respondent-level prediction.
We propose cross-survey transfer, a more rigorous evaluation framework where an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey.
Methodology
Utilizing data from the Taiwan Election and Democratization Study (TEDS) 2024, along with three open-weight LLMs (27B-120B parameters) and supervised machine learning baselines, we find that:
- Zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing within 6 percentage points (pp) of a supervised random forest trained on same-population data;
- A stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty;
- Variance collapse and safety alignment effects—two commonly cited LLM limitations—are found to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families.
These findings clarify both the promise and boundaries of silicon sampling.
Blogger's Review: The proposed cross-survey transfer evaluation framework provides a more rigorous validation method for silicon sampling, highlighting both the potential and limitations of LLMs in simulating human surveys. Further exploration of its application scenarios and optimization strategies is warranted.