Geometric Foundations of Tuning without Forgetting in Neural ODEs
This work offers a theoretical foundation for TwF in neural ODEs, which is incremental as it builds on prior work to provide exact guarantees.
The paper tackles the problem of sequential training in neural ODEs by proving that the Tuning without Forgetting (TwF) principle corresponds to a continuation along a Banach submanifold, providing an exact theoretical foundation beyond first-order approximation.
In our earlier work, we introduced the principle of Tuning without Forgetting (TwF) for sequential training of neural ODEs, where training samples are added iteratively and parameters are updated within the subspace of control functions that preserves the end-point mapping at previously learned samples on the manifold of output labels in the first-order approximation sense. In this letter, we prove that this parameter subspace forms a Banach submanifold of finite codimension under nonsingular controls, and we characterize its tangent space. This reveals that TwF corresponds to a continuation/deformation of the control function along the tangent space of this Banach submanifold, providing a theoretical foundation for its mapping-preserving (not forgetting) during the sequential training exactly, beyond first-order approximation.