Anamnesic Neural Differential Equations with Orthogonal Polynomial Projections
This addresses a memory limitation in Neural ODEs for time series analysis, offering an incremental improvement for applications like irregularly sampled data modeling.
The paper tackled the problem of Neural ODEs losing memory of past data in dynamical systems, proposing PolyODE with orthogonal polynomial projections to retain global information, which outperformed previous methods in reconstruction and prediction tasks.
Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data. These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wise integration of the latent process between observations can result in a loss of memory on the dynamic patterns of previously observed data points. We propose PolyODE, a Neural ODE that models the latent continuous-time process as a projection onto a basis of orthogonal polynomials. This formulation enforces long-range memory and preserves a global representation of the underlying dynamical system. Our construction is backed by favourable theoretical guarantees and in a series of experiments, we demonstrate that it outperforms previous works in the reconstruction of past and future data, and in downstream prediction tasks.