STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases
This addresses the problem of limited benchmarks for complex analytical reasoning in text-to-SQL tasks, benefiting analysts working with specialized relational databases, but it is incremental as it builds on existing semantic parsing methods.
The authors tackled the lack of datasets for complex analytical reasoning in question answering over structured databases by introducing STARQA, a human-created dataset on specialized domains, and found that a hybrid SQL-Python approach outperforms SQL alone, though it remains challenging for state-of-the-art LLMs.
Semantic parsing methods for converting text to SQL queries enable question answering over structured data and can greatly benefit analysts who routinely perform complex analytics on vast data stored in specialized relational databases. Although several benchmarks measure the abilities of text to SQL, the complexity of their questions is inherently limited by the level of expressiveness in query languages and none focus explicitly on questions involving complex analytical reasoning which require operations such as calculations over aggregate analytics, time series analysis or scenario understanding. In this paper, we introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized-domain databases. In addition to generating SQL directly using LLMs, we evaluate a novel approach (Text2SQLCode) that decomposes the task into a combination of SQL and Python: SQL is responsible for data fetching, and Python more naturally performs reasoning. Our results demonstrate that identifying and combining the abilities of SQL and Python is beneficial compared to using SQL alone, yet the dataset still remains quite challenging for the existing state-of-the-art LLMs.