AnnotatedTables: A Large Tabular Dataset with Language Model Annotations
This addresses the problem of scalable annotation for tabular data in machine learning applications, though it is incremental as it builds on existing LLM capabilities.
The authors tackled the scalability bottleneck of human-annotated tabular data by using language models to automatically generate annotations, resulting in the release of AnnotatedTables, a dataset of 32,119 databases with 405,616 valid SQL programs, which is the largest SQL dataset with executable queries. They demonstrated the method's utility through studies showing LLMs can translate SQL to Rel programs with few-shot learning and that TabPFN performs comparably to AutoML on annotated tables.
Tabular data is ubiquitous in real-world applications and abundant on the web, yet its annotation has traditionally required human labor, posing a significant scalability bottleneck for tabular machine learning. Our methodology can successfully annotate a large amount of tabular data and can be flexibly steered to generate various types of annotations based on specific research objectives, as we demonstrate with SQL annotation and input-target column annotation as examples. As a result, we release AnnotatedTables, a collection of 32,119 databases with LLM-generated annotations. The dataset includes 405,616 valid SQL programs, making it the largest SQL dataset with associated tabular data that supports query execution. To further demonstrate the value of our methodology and dataset, we perform two follow-up research studies. 1) We investigate whether LLMs can translate SQL programs to Rel programs, a database language previously unknown to LLMs, while obtaining the same execution results. Using our Incremental Prompt Engineering methods based on execution feedback, we show that LLMs can produce adequate translations with few-shot learning. 2) We evaluate the performance of TabPFN, a recent neural tabular classifier trained on Bayesian priors, on 2,720 tables with input-target columns identified and annotated by LLMs. On average, TabPFN performs on par with the baseline AutoML method, though the relative performance can vary significantly from one data table to another, making both models viable for practical applications depending on the situation. Our findings underscore the potential of LLMs in automating the annotation of large volumes of diverse tabular data.