Subjective judgments such as offensiveness vary widely across individuals, motivating pluralistic modeling that incorporates annotators' socio‑demographic attributes to capture this variation.
This work investigates three questions: when and where demographic information benefits pluralistic offensiveness modeling, what role explicit reasoning plays in exploiting this information, and how it compares to direct conditioning.
Experiments on three multi‑annotator datasets using prompted and fine‑tuned large language models (LLMs) reveal that demographic gains are modest and inconsistent at the single‑instance level, but become clearer when predictions are aggregated at group or population levels. Aggregation dampens idiosyncratic noise, allowing small systematic demographic signals to emerge.
We adopt a reasoning‑and‑prediction format that decomposes each judgment into an overall baseline plus a demographic‑specific deviation. Results show that both components contribute to improvement: demographics improve baseline estimation and enable meaningful shifts. Intervention studies confirm that better component estimates lead to superior final predictions.
Comparing direct demographic conditioning with explicit reasoning about how text interacts with annotator demographics, we find that reasoning yields more accurate baseline and shift estimates as well as overall predictions, whereas conditioning alone rarely produces informative shifts.
These findings suggest that mixed evidence in prior work may stem from evaluations focused solely on individual predictions. The utility of socio‑demographic information depends on both the evaluation granularity and how models are guided to use the information.
Review