Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets
For researchers and practitioners in data analytics, this work provides a benchmark of LLM capabilities for dataset-based QA, highlighting trade-offs between model size and performance.
This paper evaluates large language models (LLMs) on question answering over datasets, testing direct answering and SQL generation. Results show strong performance for large LLMs but limitations for smaller, cost-efficient models.
This paper investigates the effectiveness of large language models (LLMs) in answering questions over datasets. We examine their performance in two scenarios: (a) directly answering questions given a dataset file as input, and (b) generating SQL queries to answer questions given the schema of a relational database. We also evaluate the impact of different prompting strategies on model performance. The study includes both state-of-the-art LLMs and smaller language models that require fewer resources and operate at lower computational and financial cost. Experiments are conducted on two datasets containing questions of varying difficulty. The results demonstrate the strong performance of large LLMs, while highlighting the limitations of smaller, more cost-efficient models. These findings contribute to a better understanding of how LLMs can be utilized in data analytics tasks and their associated limitations.