Archtree: on-the-fly tree-structured exploration for latency-aware pruning of deep neural networks
This work addresses the challenge of deploying computationally intensive DNNs on resource-constrained edge devices, offering an incremental improvement over existing pruning techniques.
The paper tackles the problem of latency-aware structured pruning of deep neural networks for efficient inference on edge devices by proposing Archtree, a method that explores multiple pruned sub-models in parallel with on-the-fly latency estimation, resulting in better accuracy preservation and latency budget fitting compared to state-of-the-art methods.
Deep neural networks (DNNs) have become ubiquitous in addressing a number of problems, particularly in computer vision. However, DNN inference is computationally intensive, which can be prohibitive e.g. when considering edge devices. To solve this problem, a popular solution is DNN pruning, and more so structured pruning, where coherent computational blocks (e.g. channels for convolutional networks) are removed: as an exhaustive search of the space of pruned sub-models is intractable in practice, channels are typically removed iteratively based on an importance estimation heuristic. Recently, promising latency-aware pruning methods were proposed, where channels are removed until the network reaches a target budget of wall-clock latency pre-emptively estimated on specific hardware. In this paper, we present Archtree, a novel method for latency-driven structured pruning of DNNs. Archtree explores multiple candidate pruned sub-models in parallel in a tree-like fashion, allowing for a better exploration of the search space. Furthermore, it involves on-the-fly latency estimation on the target hardware, accounting for closer latencies as compared to the specified budget. Empirical results on several DNN architectures and target hardware show that Archtree better preserves the original model accuracy while better fitting the latency budget as compared to existing state-of-the-art methods.