$S^{2}$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
This addresses the problem of computational efficiency and model complexity in deep learning for researchers and practitioners, though it appears incremental as it builds on existing LBI methods.
The paper tackles efficient deep learning training by proposing the $S^{2}$-LBI algorithm, which achieves structural sparsity and allows network simplification or boosting, resulting in high accuracy with minimal parameters, such as 98.40% on MNIST with only 1.5K parameters.
This paper proposes a novel Stochastic Split Linearized Bregman Iteration ($S^{2}$-LBI) algorithm to efficiently train the deep network. The $S^{2}$-LBI introduces an iterative regularization path with structural sparsity. Our $S^{2}$-LBI combines the computational efficiency of the LBI, and model selection consistency in learning the structural sparsity. The computed solution path intrinsically enables us to enlarge or simplify a network, which theoretically, is benefited from the dynamics property of our $S^{2}$-LBI algorithm. The experimental results validate our $S^{2}$-LBI on MNIST and CIFAR-10 dataset. For example, in MNIST, we can either boost a network with only 1.5K parameters (1 convolutional layer of 5 filters, and 1 FC layer), achieves 98.40\% recognition accuracy; or we simplify $82.5\%$ of parameters in LeNet-5 network, and still achieves the 98.47\% recognition accuracy. In addition, we also have the learning results on ImageNet, which will be added in the next version of our report.