Sparse Flows: Pruning Continuous-depth Models
This work addresses efficiency and generalization issues in continuous-depth models for machine learning practitioners, but it is incremental as it applies pruning to existing architectures.
The authors tackled the problem of understanding and improving continuous-depth models like neural ODEs by pruning their architectures, resulting in up to 98% parameter reduction without accuracy loss and improved generalization by avoiding mode-collapse and flattening the loss surface.
Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.