From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models
This addresses the problem of improving predictive modeling in industries like healthcare and finance by enabling LLMs to better handle tabular data, representing a new paradigm rather than an incremental advance.
The paper tackles the challenge of developing universal models for tabular data that can transfer knowledge and generalize from limited data by introducing Generative Tabular Learning (GTL), a framework integrating large language models (LLMs) into tabular deep learning, which demonstrates superior zero-shot and in-context learning capabilities across 384 datasets, outperforming traditional methods and rivaling GPT-4 in some cases without fine-tuning.
Tabular data is foundational to predictive modeling in various crucial industries, including healthcare, finance, retail, sustainability, etc. Despite the progress made in specialized models, there is an increasing demand for universal models that can transfer knowledge, generalize from limited data, and follow human instructions. These are challenges that current tabular deep learning approaches have not fully tackled. Here we introduce Generative Tabular Learning (GTL), a novel framework that integrates the advanced functionalities of large language models (LLMs)-such as prompt-based zero-shot generalization and in-context learning-into tabular deep learning. GTL capitalizes on the pre-training of LLMs on diverse tabular data, enhancing their understanding of domain-specific knowledge, numerical sequences, and statistical dependencies critical for accurate predictions. Our empirical study spans 384 public datasets, rigorously analyzing GTL's convergence and scaling behaviors and assessing the impact of varied data templates. The GTL-enhanced LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks. Notably, it achieves this without fine-tuning, outperforming traditional methods and rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we not only foster a deeper integration of LLMs' sophisticated abilities into tabular data comprehension and application but also offer a new training resource and a test bed for LLMs to enhance their ability to comprehend tabular data. To facilitate reproducible research, we release our code, data, and model checkpoints at https://github.com/microsoft/Industrial-Foundation-Models.