On the Efficiency of NLP-Inspired Methods for Tabular Deep Learning
This addresses efficiency concerns for researchers and practitioners using tabular deep learning, but it is incremental as it builds on existing methods.
The paper tackles the problem of computational inefficiency in tabular deep learning models, which have grown large due to NLP-inspired methods, and finds that focusing on efficiency can maintain performance while reducing resource use.
Recent advancements in tabular deep learning (DL) have led to substantial performance improvements, surpassing the capabilities of traditional models. With the adoption of techniques from natural language processing (NLP), such as language model-based approaches, DL models for tabular data have also grown in complexity and size. Although tabular datasets do not typically pose scalability issues, the escalating size of these models has raised efficiency concerns. Despite its importance, efficiency has been relatively underexplored in tabular DL research. This paper critically examines the latest innovations in tabular DL, with a dual focus on performance and computational efficiency. The source code is available at https://github.com/basf/mamba-tabular.