Are Data Embeddings effective in time series forecasting?
This work challenges a common practice in time series forecasting, potentially simplifying model design and reducing computational costs for researchers and practitioners.
The paper investigates the effectiveness of data embedding layers in time series forecasting models, finding through ablation studies that removing these layers often improves accuracy and computational efficiency without degrading performance, with gains exceeding typical reported differences between models.
Time series forecasting plays a crucial role in many real-world applications, and numerous complex forecasting models have been proposed in recent years. Despite their architectural innovations, most state-of-the-art models report only marginal improvements -- typically just a few thousandths in standard error metrics. These models often incorporate complex data embedding layers to transform raw inputs into higher-dimensional representations to enhance accuracy. But are data embedding techniques actually effective in time series forecasting? Through extensive ablation studies across fifteen state-of-the-art models and four benchmark datasets, we find that removing data embedding layers from many state-of-the-art models does not degrade forecasting performance. In many cases, it improves both accuracy and computational efficiency. The gains from removing embedding layers often exceed the performance differences typically reported between competing models. Code available at: https://github.com/neuripsdataembedidng/DataEmbedding