eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations
This work addresses the need for more efficient and flexible AI inference in resource-constrained edge environments, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of inefficient AI inference on edge devices by introducing the eIQ Neutron NPU with a co-designed compiler, achieving an average speedup of 1.8x (up to 4x peak) compared to leading embedded solutions at equal resources and up to 3.3x higher performance against NPUs with double the resources.
Neural Processing Units (NPUs) are key to enabling efficient AI inference in resource-constrained edge environments. While peak tera operations per second (TOPS) is often used to gauge performance, it poorly reflects real-world performance and typically rather correlates with higher silicon cost. To address this, architects must focus on maximizing compute utilization, without sacrificing flexibility. This paper presents the eIQ Neutron efficient-NPU, integrated into a commercial flagship MPU, alongside co-designed compiler algorithms. The architecture employs a flexible, data-driven design, while the compiler uses a constrained programming approach to optimize compute and data movement based on workload characteristics. Compared to the leading embedded NPU and compiler stack, our solution achieves an average speedup of 1.8x (4x peak) at equal TOPS and memory resources across standard AI-benchmarks. Even against NPUs with double the compute and memory resources, Neutron delivers up to 3.3x higher performance.