Exact-Match Scoring Rejects 75% of Correct Text-to-SQL Queries Once AI Operators Are in the SQL

arXiv 2609.21133·medium signalXBlueskyLinkedInCopy link

SQL augmented with AI operators produces non-deterministic results that break execution-accuracy scoring, and the authors measure the damage: traditional Execution Accuracy detects as few as 25% of correct translations, while a state-of-the-art LLM autorater falsely rejects 32% of accurate queries because it must judge relational logic and AI semantics at once. Their multilayered framework validates the deterministic database logic separately from the flexible AI operations and reaches up to 97.2% overall accuracy across both BigQuery and the academic ThalamusDB system. For anyone benchmarking a text-to-SQL agent against a warehouse with LLM functions, this says the metric, not the model, is likely the thing producing bad numbers.