Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models
This provides theoretical grounding for a key component in deep learning models that handle long sequences, addressing a gap in the literature.
The paper tackled the mathematical foundations of the singular HiPPO-LegS ODE used in deep state space models, establishing that it is well-posed and that numerical discretization schemes converge for Riemann integrable input functions.
In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions.