oneDNN Graph Compiler: A Hybrid Approach for High-Performance Deep Learning Compilation
This work addresses the problem of inefficient deep learning compilation for AI hardware, offering a solution that improves performance for developers and users of DNN models, though it appears incremental as it builds on existing compiler and kernel techniques.
The paper tackles the challenge of achieving high-performance deep learning compilation across entire neural network graphs, rather than just compute-intensive operations, by introducing a hybrid compiler that combines compiler optimization with expert-tuned kernels, resulting in significant performance gains over existing tensor compilers and primitives libraries on Intel Xeon Scalable Processors.
With the rapid development of deep learning models and hardware support for dense computing, the deep learning workload characteristics changed significantly from a few hot spots on compute-intensive operations to a broad range of operations scattered across the models. Accelerating a few compute-intensive operations using the expert-tuned implementation of primitives does not fully exploit the performance potential of AI hardware. Various efforts have been made to compile a full deep neural network (DNN) graph. One of the biggest challenges is to achieve high-performance tensor compilation by generating expert level performance code for the dense compute-intensive operations and applying compilation optimization at the scope of DNN computation graph across multiple compute-intensive operations. We present oneDNN Graph Compiler, a tensor compiler that employs a hybrid approach of using techniques from both compiler optimization and expert-tuned kernels for high performance code generation of the deep neural network graph. oneDNN Graph Compiler addresses unique optimization challenges in the deep learning domain, such as low-precision computation, aggressive fusion of graph operations, optimization for static tensor shapes and memory layout, constant weight optimization, and memory buffer reuse. Experimental results demonstrate significant performance gains over existing tensor compiler and primitives library for performance-critical DNN computation graphs and end-to-end models on Intel Xeon Scalable Processors.