This paper investigates whether high annotation reproducibility necessarily implies that an LLM‑inferred measure captures the intended construct. We paired structured survey responses with free‑text submissions from the same stakeholders in the European Commission’s AI Act consultation dataset.
LLM annotations of the consultation texts are highly reproducible, with an intraclass correlation of $\text{ICC}=0.99$, yet they show limited convergence with the nominal construct measured by the survey.
The divergence between survey‑based and LLM‑derived text measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text‑based consultations than in survey responses, with a divergence of approximately $\Delta = +1.0$, whereas public authorities and several non‑business groups exhibit smaller or negative divergences.
Spatial analysis of the text scores reveals positive spatial autocorrelation across European countries, with Moran’s I $\text{I}=0.347$ ($p=0.036$), indicating that stakeholders from neighboring countries tend to hold more similar text‑based stances on AI safety.
Despite these divergences, survey‑reported concerns remain strongly associated with support for explainability across all levels of divergence.
The findings demonstrate that high LLM annotation reproducibility can coexist with poor construct correspondence, motivating validation procedures that distinguish reproducibility, construct validity, and communication‑context variation when LLMs are employed as measurement instruments.
Review: The study cautions researchers against equating reproducibility with validity in LLM‑based text measurement and underscores the need for cross‑context construct validation.