Evaluating the correctness of AI-generated SQL queries hinges on determining whether a query is correct—meaning it executes to the same result as a human-written reference. We investigate which signals predict correctness on challenging multi-table text-to-SQL tasks, using AUROC to measure how well each ranks correct queries above incorrect ones.
On the BIRD and Spider datasets, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between approximately 0.61 and 0.68 AUROC, with string self-consistency being the strongest at 0.675; white-box log-probability is similar at around 0.67. The signals that surpass this ceiling are verification-based: LLM judges score between 0.72 (GPT-4o-mini) and 0.78 (Claude). Different providers’ judges make different types of errors, so a two-provider ensemble achieves an AUROC of 0.82 with well-calibrated probabilities (expected calibration error of 0.03) and supports useful abstention frontiers (for instance, answering 27% of questions at a 24% selective risk), where self-consistency offers no valid low-risk subset.
This pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, achieve around 0.77 to 0.79 AUROC in-distribution but drop to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL resides in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently requires a large frozen reasoning model.
Blogger's Review: This paper provides an in-depth analysis of the predictive capabilities of various signals regarding query correctness in multi-table text-to-SQL transformations, particularly highlighting the advantages of verification-based signals. It underscores the limitations and potential of current AI models in handling complex queries, suggesting future research could explore enhancing the generalization capabilities of verifiers across schemas.