Latent Neural ODEs with Sparse Bayesian Multiple Shooting
This addresses the challenge of training dynamic models like neural ODEs for practitioners, offering a more stable and efficient alternative to heuristic methods.
The paper tackled the problem of training neural ODEs on long trajectories by proposing a principled multiple shooting technique that splits trajectories into segments optimized in parallel with probabilistic continuity control, resulting in efficient and stable training and state-of-the-art performance on large-scale benchmarks.
Training dynamic models, such as neural ODEs, on long trajectories is a hard problem that requires using various tricks, such as trajectory splitting, to make model training work in practice. These methods are often heuristics with poor theoretical justifications, and require iterative manual tuning. We propose a principled multiple shooting technique for neural ODEs that splits the trajectories into manageable short segments, which are optimised in parallel, while ensuring probabilistic control on continuity over consecutive segments. We derive variational inference for our shooting-based latent neural ODE models and propose amortized encodings of irregularly sampled trajectories with a transformer-based recognition network with temporal attention and relative positional encoding. We demonstrate efficient and stable training, and state-of-the-art performance on multiple large-scale benchmark datasets.