TENT: Tensorized Encoder Transformer for Temperature Forecasting
This work addresses weather forecasting, a critical problem for science, business, and society, but it is incremental as it adapts an existing Transformer architecture with tensorial attention for a specific domain.
The authors tackled temperature forecasting by introducing TENT, a Transformer-based model with tensorial attention that processes weather data as multidimensional tensors, achieving better performance than classical Transformers, 3D CNNs, LSTM, and Convolutional LSTM on two real-life datasets from the USA, Canada, and Europe.
Reliable weather forecasting is of great importance in science, business, and society. The best performing data-driven models for weather prediction tasks rely on recurrent or convolutional neural networks, where some of which incorporate attention mechanisms. In this work, we introduce a novel model based on Transformer architecture for weather forecasting. The proposed Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention and thus it exploits the spatiotemporal structure of weather data by processing it in multidimensional tensorial format. We show that compared to the classical encoder transformer, 3D convolutional neural networks, LSTM, and Convolutional LSTM, the proposed TENT model can better learn the underlying complex pattern of the weather data for the studied temperature prediction task. Experiments on two real-life weather datasets are performed. The datasets consist of historical measurements from weather stations in the USA, Canada and Europe. The first dataset contains hourly measurements of weather attributes for 30 cities in the USA and Canada from October 2012 to November 2017. The second dataset contains daily measurements of weather attributes of 18 cities across Europe from May 2005 to April 2020. Two attention scores are introduced based on the obtained tonsorial attention and are visualized in order to shed light on the decision-making process of our model and provide insight knowledge on the most important cities for the target cities.