NeFut Logo NeFut
中 Admin Login

[CS.AI] Counting Under Disagreement: Dollar‑Weighted Measurement, Selective Machine Labelling and Budgeted Human Review

Published at: 2026-11-11 22:00 Last updated: 2026-11-12 00:51
#AI #Machine Learning #optimization

We investigate AI‑assisted measurement when a single text label applies to many administrative records, the target is a monetary total, and human readings that anchor the machines disagree. The testbed consists of 121,070 grant‑purpose texts from US tax filings (799,675 records, $126.4\text{B}$), labelled by a rule‑based classifier and three language models, with a designed sample of 651 texts read independently by three people under a common codebook. Exact conditions are stated for a fixed‑predictor difference estimator and for a cross‑fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and the implemented fixed‑size procedures are assessed by simulation: eight predictors whose own totals span $11.2\text{B}$‑$15.0\text{B}$ agree within $0.4\text{B}$ after rectification. Two rules that accept machine labels at a stated dollar‑weighted error—weighted conformal p‑values under a dollar measure and Learn‑then‑Test on the design error estimate—are evaluated on nested held‑out folds and on redraws of the design with synthetic truths; their error control in deployment is not established, and the count‑based rule accepts no text under a dollar‑drawn sample. In a replay that charges every human label, a two‑phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar‑proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor. Independent readers shift the estimated total by $3.93\text{B}$, 2.2 times the within‑reader sampling interval, and institution‑level intervals under‑cover at every size. Every headline result is re‑run under each reader. Review: The work highlights the promise of learned predictions for guiding human verification, yet also exposes the difficulty of guaranteeing error control when annotators disagree and budgets are limited.

Original Source: https://arxiv.org/abs/2611.13215

[h] Back to Home