Time Regularization in Optimal Time Variable Learning
This work addresses efficiency and complexity issues in deep learning for researchers and practitioners, but it is incremental as it builds on prior optimal time variable learning concepts.
The paper tackles the problem of optimizing time variable learning in deep neural networks by introducing a regularization term for time horizons and an adaptive pruning approach for ResNets, resulting in reduced network complexity and training time while maintaining expressiveness, as demonstrated on MNIST and Fashion MNIST datasets.
Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning.