Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting
This work addresses the challenge of applying Transformers to tabular data, a domain where gradient boosting traditionally excels, offering a significant performance improvement for practitioners in forecasting.
This paper introduces a discrete tokenization method that enables Transformers to achieve state-of-the-art performance in tabular forecasting. The approach, which combines discrete tokenization with Gaussian smoothing, outperforms tuned XGBoost by 10.8% in median MAE on a dataset of 600K entities and 5M training examples, while also achieving a low Kolmogorov-Smirnov (KS) score of 0.0045 for calibration.
Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.