TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023
This work addresses the challenge of making deep learning competitive with non-DL methods like GBDT for tabular data problems, offering a more efficient and high-performing solution for researchers and practitioners in machine learning.
The authors tackled the problem of improving deep learning models for tabular data by introducing TabR, a feed-forward network with a k-nearest-neighbors-like component, which achieved state-of-the-art performance on several benchmarks and outperformed gradient-boosted decision trees on a 'GBDT-friendly' benchmark.
Deep learning (DL) models for tabular data problems (e.g. classification, regression) are currently receiving increasingly more attention from researchers. However, despite the recent efforts, the non-DL algorithms based on gradient-boosted decision trees (GBDT) remain a strong go-to solution for these problems. One of the research directions aimed at improving the position of tabular DL involves designing so-called retrieval-augmented models. For a target object, such models retrieve other objects (e.g. the nearest neighbors) from the available training data and use their features and labels to make a better prediction. In this work, we present TabR -- essentially, a feed-forward network with a custom k-Nearest-Neighbors-like component in the middle. On a set of public benchmarks with datasets up to several million objects, TabR marks a big step forward for tabular DL: it demonstrates the best average performance among tabular DL models, becomes the new state-of-the-art on several datasets, and even outperforms GBDT models on the recently proposed "GBDT-friendly" benchmark (see Figure 1). Among the important findings and technical details powering TabR, the main ones lie in the attention-like mechanism that is responsible for retrieving the nearest neighbors and extracting valuable signal from them. In addition to the much higher performance, TabR is simple and significantly more efficient compared to prior retrieval-based tabular DL models.