Taylorformer: Probabilistic Modelling for Random Processes including Time Series
This addresses the need for uncertainty-aware predictions in time series and related domains, representing a novel method for a known bottleneck.
The paper tackles the problem of probabilistic modeling for random processes like time series by proposing the Taylorformer, which outperforms state-of-the-art methods with at least a 14% MSE improvement on forecasting tasks and better log-likelihood on 5/6 Neural Process tasks.
We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material.