Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws
This addresses a critical limitation in AI for science and engineering by clarifying the extrapolation gap, though it is incremental as it builds on existing issues without presenting a new solution.
The paper tackles the problem of why neural networks, particularly Foundation Models for time series, fail at extrapolation or long-range forecasting, identifying a fundamental property that explains performance deterioration outside training domains, with empirical results highlighting implications for current architectures.
Motivated by the remarkable success of Foundation Models (FMs) in language modeling, there has been growing interest in developing FMs for time series prediction, given the transformative power such models hold for science and engineering. This culminated in significant success of FMs in short-range forecasting settings. However, extrapolation or long-range forecasting remains elusive for FMs, which struggle to outperform even simple baselines. This contrasts with physical laws which have strong extrapolation properties, and raises the question of the fundamental difference between the structure of neural networks and physical laws. In this work, we identify and formalize a fundamental property characterizing the ability of statistical learning models to predict more accurately outside of their training domain, hence explaining performance deterioration for deep learning models in extrapolation settings. In addition to a theoretical analysis, we present empirical results showcasing the implications of this property on current deep learning architectures. Our results not only clarify the root causes of the extrapolation gap but also suggest directions for designing next-generation forecasting models capable of mastering extrapolation.