TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns
This addresses a practical challenge in healthcare data analysis where column mismatches occur, though it is incremental as it builds on existing pre-training methods.
The authors tackled the problem of applying pre-trained Transformer models to tabular data with unseen columns in downstream tasks, and TabRet achieved the best AUC performance on four healthcare datasets.
We present \emph{TabRet}, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called \emph{retokenizing}, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains. The code is available at https://github.com/pfnet-research/tabret .