EnSyth: A Pruning Approach to Synthesis of Deep Learning Ensembles
This work addresses the challenge of deploying efficient deep learning models in strict latency or resource-constrained applications, but it is incremental as it builds on existing pruning and ensemble methods.
The paper tackles the problem of maintaining accuracy in compressed deep neural networks for resource-limited environments by proposing EnSyth, a pruning-based ensemble approach that synthesizes diverse compressed models, resulting in improved predictability over baseline models on CIFAR-10 and CIFAR-5 datasets with LeNet-5.
Deep neural networks have achieved state-of-art performance in many domains including computer vision, natural language processing and self-driving cars. However, they are very computationally expensive and memory intensive which raises significant challenges when it comes to deploy or train them on strict latency applications or resource-limited environments. As a result, many attempts have been introduced to accelerate and compress deep learning models, however the majority were not able to maintain the same accuracy of the baseline models. In this paper, we describe EnSyth, a deep learning ensemble approach to enhance the predictability of compact neural network's models. First, we generate a set of diverse compressed deep learning models using different hyperparameters for a pruning method, after that we utilise ensemble learning to synthesise the outputs of the compressed models to compose a new pool of classifiers. Finally, we apply backward elimination on the generated pool to explore the best performing combinations of models. On CIFAR-10, CIFAR-5 data-sets with LeNet-5, EnSyth outperforms the predictability of the baseline model.