Language‑model agents are increasingly employed as synthetic respondents in surveys and social simulations, yet their realism is often judged solely by population‑level averages or distributional similarity. We evaluated Claude, Codex and Kimi under four predefined prompt designs against matched farmer decisions from China and four African countries. Some configurations reproduced observed means and adoption rates, but their person‑level predictions were weak: decisions clustered around typical values and policy‑relevant extremes were largely absent. Surprisingly, a simple generator fitted only to the observed marginal distribution achieved higher distributional similarity than any language‑model configuration. Adding prompts yielded conditional gains rather than universal improvement—results varied with model, outcome, population, and validation target. We term this the “average‑farmer illusion”: a synthetic population may appear realistic while failing to reproduce who does what and how behavior varies. To address this, we provide a claim‑matched validation framework and reusable modular prompts that make prompt construction auditable. Population‑level resemblance should be treated as the start of validation, not as evidence of accurate individual simulation.
Review