LLM‑based social simulation offers a complementary approach to traditional surveys and behavioral experiments. The key challenge is how to measure how faithfully a model reproduces human behavior and how to steer the model toward higher fidelity.
The prevailing practice evaluates models by accuracy: checking whether the model outputs the single response observed from a human and then training the LLM to reproduce that hard label. Human behavior, however, is inherently subjective— the same person in the same situation may reasonably act differently. An observed response is merely one draw from an underlying response distribution, making accuracy‑based evaluation unreliable and hard‑label training misleading for subjective tasks.
To quantify the degree of subjectivity, the authors introduce the subjectivity coefficient, defined as the entropy of the response distribution:
$$ S = -\sum_{i} p_i \log p_i $$
where $p_i$ is the probability of the $i$‑th possible response. Larger $S$ indicates a more subjective task; $S\\approx 0$ corresponds to near‑objective tasks such as code generation.
Guided by the subjectivity coefficient, they propose Subjectivity‑Adaptive soft‑Label Training (SALT). SALT works as follows:
- For each input $x$, compute its subjectivity coefficient $S(x)$ and derive a semantic neighbourhood radius $r$.
- Gather all observed outputs within this neighbourhood and form a soft‑label distribution $\tilde{p}(y|x)$.
- Train the model with cross‑entropy against the soft label.
When $S(x)\rightarrow 0$, the neighbourhood collapses, $\tilde{p}$ becomes a single hard label, and SALT naturally reduces to standard training.
Because most public datasets only contain a single observed response per input, the authors build a new benchmark SUBJSIM: 19,300 contexts, 193 annotators, and 100 subjective questions. In experiments, models are trained only on the single observed output per input, but evaluation compares the model’s predictions against the full response distributions, demonstrating feasibility in realistic settings.
Results on SUBJSIM show that SALT consistently outperforms hard‑label baselines across tasks with varying subjectivity levels, with the most pronounced gains on highly subjective items.
Blogger's Review: The paper formalizes task subjectivity with an entropy‑based metric and introduces an adaptive soft‑label training regime that directly addresses the shortcomings of accuracy‑only evaluation. The newly released SUBJSIM dataset is a valuable resource, and the proposed method’s generality invites further exploration in other LLM‑driven social‑science applications.