Evaluating Cross-Domain Text-to-SQL Models and Benchmarks
Mohammadreza Pourreza Davood Rafiei
Text-to-SQL benchmarks play a crucial role in evaluating the progress made in
the field and the ranking of different models. However, accurately matching a
model-generated SQL query to a reference SQL query in a benchmark fails for
various reasons, such as underspecified natural language queries, inherent
assumptions in both model-generated and reference queries, and the
non-deterministic nature of SQL output under certain conditions. In this paper,
we conduct an extensive study of several prominent cross-domain text-to-SQL
benchmarks and re-evaluate some of the top-performing models within these
benchmarks, by both manually evaluating the SQL queries and rewriting them in
equivalent expressions. Our evaluation reveals that attaining a perfect
performance on these benchmarks is unfeasible due to the multiple
interpretations that can be derived from the provided samples. Furthermore, we
find that the true performance of the models is underestimated and their
relative performance changes after a re-evaluation. Most notably, our
evaluation reveals a surprising discovery: a recent GPT4-based model surpasses
the gold standard reference queries in the Spider benchmark in our human
evaluation. This finding highlights the importance of interpreting benchmark
evaluations cautiously, while also acknowledging the critical role of
additional independent evaluations in driving advancements in the field.