Table-GPT: Table-tuned GPT for Diverse Table Tasks
This work addresses the limitation of language models in handling two-dimensional table data, which is important for applications involving structured data analysis, but it is incremental as it builds on existing models with fine-tuning.
The paper tackles the problem of language models being sub-optimal for table-related tasks due to their one-dimensional text pre-training, and proposes a 'table-tuning' paradigm that fine-tunes models like GPT-3.5 and ChatGPT on synthesized table tasks, resulting in Table-GPT models that consistently outperform vanilla versions on a wide range of table tasks and show strong generalizability to new instructions.
Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable abilities to follow diverse human instructions and perform a wide range of tasks. However, when probing language models using a range of basic table-understanding tasks, we observe that today's language models are still sub-optimal in many table-related tasks, likely because they are pre-trained predominantly on \emph{one-dimensional} natural-language texts, whereas relational tables are \emph{two-dimensional} objects. In this work, we propose a new "\emph{table-tuning}" paradigm, where we continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using diverse table-tasks synthesized from real tables as training data, with the goal of enhancing language models' ability to understand tables and perform table tasks. We show that our resulting Table-GPT models demonstrate (1) better \emph{table-understanding} capabilities, by consistently outperforming the vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout unseen tasks, and (2) strong \emph{generalizability}, in its ability to respond to diverse human instructions to perform new table-tasks, in a manner similar to GPT-3.5 and ChatGPT.