Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices
This work addresses deployment challenges for low-resource devices, but it is incremental as it builds on existing binary neural network architectures without retraining.
The authors tackled the problem of deploying deep neural networks on low-resource devices by proposing a method that combines binary neural networks with early-exiting strategies, achieving latency under 6ms and favorable quality-efficiency trade-offs on audio classification tasks.
Deep neural networks have significantly improved performance on a range of tasks with the increasing demand for computational resources, leaving deployment on low-resource devices (with limited memory and battery power) infeasible. Binary neural networks (BNNs) tackle the issue to an extent with extreme compression and speed-up gains compared to real-valued models. We propose a simple but effective method to accelerate inference through unifying BNNs with an early-exiting strategy. Our approach allows simple instances to exit early based on a decision threshold and utilizes output layers added to different intermediate layers to avoid executing the entire binary model. We extensively evaluate our method on three audio classification tasks and across four BNNs architectures. Our method demonstrates favorable quality-efficiency trade-offs while being controllable with an entropy-based threshold specified by the system user. It also results in better speed-ups (latency less than 6ms) with a single model based on existing BNN architectures without retraining for different efficiency levels. It also provides a straightforward way to estimate sample difficulty and better understanding of uncertainty around certain classes within the dataset.