Transformers predicting the future. Applying attention in next-frame and time series forecasting
This work tackles forecasting problems in domains like video and time series, but it is incremental as it builds on existing Transformer-based solutions without introducing a fundamentally new approach.
The paper tests and evaluates Transformer-like models for time series forecasting and next-frame prediction, focusing on addressing issues like anomaly susceptibility and space complexity through hyperparameter tuning, data preprocessing, and architectural modifications.
Recurrent Neural Networks were, until recently, one of the best ways to capture the timely dependencies in sequences. However, with the introduction of the Transformer, it has been proven that an architecture with only attention-mechanisms without any RNN can improve on the results in various sequence processing tasks (e.g. NLP). Multiple studies since then have shown that similar approaches can be applied for images, point clouds, video, audio or time series forecasting. Furthermore, solutions such as the Perceiver or the Informer have been introduced to expand on the applicability of the Transformer. Our main objective is testing and evaluating the effectiveness of applying Transformer-like models on time series data, tackling susceptibility to anomalies, context awareness and space complexity by fine-tuning the hyperparameters, preprocessing the data, applying dimensionality reduction or convolutional encodings, etc. We are also looking at the problem of next-frame prediction and exploring ways to modify existing solutions in order to achieve higher performance and learn generalized knowledge.