Semantic query engines ask a large language model to judge each row, and existing benchmarks at temperature 0 only compare a few executions. We re‑executed the text and tabular workloads of SemBench, LRO‑Bench and UDA‑Bench, logged about 1.2 million model calls, and re‑analyzed SemBench’s published runs.
At temperature 0, the same model changes the operator output on 0.07%‑10.2% of rows across five independent runs, with a worst‑case task reaching 24.0%. Answer plans amplify these changes exponentially, as a possible‑worlds model predicts: on UDA‑Bench only a few percent of extracted values differ, but roughly two‑thirds of full answers do.
SemBench’s published results show that 21% of text results vary in quality between runs, even for engines with deterministic plans. Several per‑query system comparisons and 16 of 28 LRO‑Bench implementation recommendations remain inconclusive given the number of benchmark executions.
“Changing rows” are rows where the model disagrees with itself; intra‑run disagreement occurs as often as inter‑run. The decision margin or reasoning length of a run predicts which rows will change. Voting reduces variance but does not eliminate it, and a calibrated cascade retains its guarantee while the rows it returns still vary.
From these findings we derive reporting rules for benchmarks and design rules for engines, requiring variance statistics in reports and variance‑mitigation mechanisms in implementations.
Review