The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling
This work addresses tabular modeling for machine learning practitioners, but it is incremental as it builds on an existing state-of-the-art method.
The paper tackled the problem of improving deep learning for tabular data by proposing modifications to the TabTransformer architecture, resulting in over 1% AUROC gains on binary classification tasks across three datasets.
There is an increasing interest in the application of deep learning architectures to tabular data. One of the state-of-the-art solutions is TabTransformer which incorporates an attention mechanism to better track relationships between categorical features and then makes use of a standard MLP to output its final logits. In this paper we propose multiple modifications to the original TabTransformer performing better on binary classification tasks for three separate datasets with more than 1% AUROC gains. Inspired by gated MLP, linear projections are implemented in the MLP block and multiple activation functions are tested. We also evaluate the importance of specific hyper parameters during training.