TableLlama: Towards Open Large Generalist Models for Tables
This addresses the need for open-source, general-purpose models to handle various table tasks without task-specific designs, benefiting researchers and practitioners in data analysis and AI.
The paper tackles the problem of developing generalist models for diverse table-based tasks by creating TableInstruct, a dataset for instruction tuning, and TableLlama, an open-source model fine-tuned from Llama 2. It achieves comparable or better performance than SOTA on 7 out of 8 in-domain tasks and shows 5-44 point gains on out-of-domain datasets.
Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.