This study reports the empirical reliability of Gemini models as audio judges scoring full-duplex agent conversations directly from raw stereo waveforms. We tested three models in the Gemini family: 2.5 Flash, 3.5 Flash, and 3.1 Pro. Our primary evidence base utilizes Gemini 2.5 Flash as the ground-truth model, validated against three calibrated human raters across 209 stereo sessions, rated on 8 production dimensions: 152 full-duplex conversations and 57 adversarial defect-injected clips.
Results show that Gemini 2.5 Flash is consistent across three tests: (i) On 5 of 8 dimensions, the LALM-human Spearman rho departs from the pairwise human-human rho by at most 0.07, with 95% bootstrap confidence intervals overlapping on 7 of 8 dimensions; (ii) The LALM agrees with the three-rater human mean within 1 point on 60% to 92% of sessions across 6 of 8 dimensions; (iii) On 48 (defect, dimension) cells, the LALM is as sensitive as humans or better under Newcombe-Wilson 95% confidence intervals, though most are underpowered nulls rather than demonstrated parity.
Moreover, rank-ordering ability transfers across the Gemini family: 3.5 Flash improves simple agreement to 100% on all dimensions, while 3.1 Pro rates several dimensions significantly lower than humans despite comparable rank correlation. A model swap should be re-validated on calibration specifically, not assumed from rank correlation alone. We identify four areas requiring careful deployment and estimate that human rating costs roughly two orders of magnitude more than the equivalent LALM workload. The data presented here provides a defensible empirical basis for deploying the LALM as a substitute or fourth rater on dimensions where evidence supports it.
Blogger's Review: This study provides significant empirical support for the application of LALM in voice agents, particularly in terms of cost-effectiveness. The Gemini series models demonstrate potential in evaluating full-duplex conversations compared to human raters, but careful calibration is still necessary when swapping models. Future research could further explore the applicability and limitations of different models.