Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures
This work addresses energy and memory constraints for deploying multi-task NLP models on resource-constrained edge devices, representing an incremental improvement over existing single-task optimizations.
The paper tackles the problem of high memory requirements for multi-task NLP inference on edge devices by proposing adapter-ALBERT, which enables efficient data reuse across tasks, achieving up to 40% reduction in energy consumption and 30% improvement in area efficiency compared to traditional ALBERT models.
Executing machine learning inference tasks on resource-constrained edge devices requires careful hardware-software co-design optimizations. Recent examples have shown how transformer-based deep neural network models such as ALBERT can be used to enable the execution of natural language processing (NLP) inference on mobile systems-on-chip housing custom hardware accelerators. However, while these existing solutions are effective in alleviating the latency, energy, and area costs of running single NLP tasks, achieving multi-task inference requires running computations over multiple variants of the model parameters, which are tailored to each of the targeted tasks. This approach leads to either prohibitive on-chip memory requirements or paying the cost of off-chip memory access. This paper proposes adapter-ALBERT, an efficient model optimization for maximal data reuse across different tasks. The proposed model's performance and robustness to data compression methods are evaluated across several language tasks from the GLUE benchmark. Additionally, we demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator to extrapolate performance, power, and area improvements over the execution of a traditional ALBERT model on the same hardware platform.