A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers
This provides a flexible and effective solution for event prediction in tabular data, addressing issues like cost and brittleness in existing methods, though it is incremental as it builds on standard transformer techniques.
The paper tackles the problem of predicting events in tabular data, such as fraud detection or customer ratings, by proposing a simple baseline using autoregressive transformers with basic positional embeddings and a causal language modeling objective, which outperforms existing approaches across popular datasets and can handle multiple use-cases like label prediction, imputation, and sequence modeling.
Many real-world applications of tabular data involve using historic events to predict properties of new ones, for example whether a credit card transaction is fraudulent or what rating a customer will assign a product on a retail platform. Existing approaches to event prediction include costly, brittle, and application-dependent techniques such as time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. Moreover, these approaches often assume specific use-cases, for example that we know the labels of all historic events or that we only predict a pre-specified label and not the data's features themselves. In this work, we propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases. We demonstrate that the same model can predict labels, impute missing values, or model event sequences.