Evaluating the Data Model Robustness of Text-to-SQL Systems Based on
  Real User Queries
Jonathan Fürst Catherine Kosten Farhard Nooralahzadeh Yi Zhang Kurt Stockinger
Text-to-SQL systems (also known as NL-to-SQL systems) have become an
increasingly popular solution for bridging the gap between user capabilities
and SQL-based data access. These systems translate user requests in natural
language to valid SQL statements for a specific database. Recent Text-to-SQL
systems have benefited from the rapid improvement of transformer-based language
models. However, while Text-to-SQL systems that incorporate such models
continuously reach new high scores on -- often synthetic -- benchmark datasets,
a systematic exploration of their robustness towards different data models in a
real-world, realistic scenario is notably missing. This paper provides the
first in-depth evaluation of the data model robustness of Text-to-SQL systems
in practice based on a multi-year international project focused on Text-to-SQL
interfaces. Our evaluation is based on a real-world deployment of FootballDB, a
system that was deployed over a 9 month period in the context of the FIFA World
Cup 2022, during which about 6K natural language questions were asked and
executed. All of our data is based on real user questions that were asked live
to the system. We manually labeled and translated a subset of these questions
for three different data models. For each data model, we explore the
performance of representative Text-to-SQL systems and language models. We
further quantify the impact of training data size, pre-, and post-processing
steps as well as language model inference time. Our comprehensive evaluation
sheds light on the design choices of real-world Text-to-SQL systems and their
impact on moving from research prototypes to real deployments. Last, we provide
a new benchmark dataset to the community, which is the first to enable the
evaluation of different data models for the same dataset and is substantially
more challenging than most previous datasets in terms of query complexity.