With organizations and researchers increasingly interested in using large language models (LLMs) instead of human participants for A/B testing, aiming for faster and cheaper experiments, we study when a treatment effect estimated from LLM outcomes can recover the effect measured on the target human population. Distributional equivalence between LLM and human outcomes would render any standard estimator valid, but this is unrealistic. Therefore, we developed a statistical framework that adapts surrogate endpoint theory to LLMs.
This framework shows that calibrating LLM outcomes to human outcomes identifies the average treatment effect under surrogacy and comparability conditions that are jointly weaker than distributional equivalence. When these conditions fail, the effect of interest is only partially identified, and we provide diagnostics that can falsify surrogacy on historical experiments along with a bound on the worst-case bias from limited overlap.
Moreover, we demonstrate that the stochasticity inherent to LLMs introduces both bias and variance, but using an average of multiple draws as the surrogate mitigates both. We illustrate the methods and theory in simulations and an application to A/B tests on Upworthy headlines.
A central takeaway from our work is that the validity of LLM outcomes as surrogates can only be falsified for past treatments and never verified for new ones, so human experiments remain indispensable for novel interventions. We discuss the role of LLM choice, prompting, and temperature as design variables, and how to size human experiments for validation.
Blogger's Review: This research provides crucial statistical foundations for the application of LLMs in A/B testing, highlighting the limitations of using LLM results as effective surrogates without human experimentation. While LLMs can accelerate the experimental process, caution is necessary regarding the validity of their outcomes, ensuring essential human validation for new interventions.