ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations
This addresses the performance bottleneck in deep learning inference for users of heterogeneous hardware by providing a unified optimization approach, though it is incremental as it builds on existing compiler techniques.
The paper tackles the problem of inefficient deep learning inference due to the separation of graph-level and operator-level optimizations in compilers, proposing ALT, a compiler that jointly optimizes both levels, achieving average speedups of 1.5x for single operators and 1.4x for end-to-end inference compared to state-of-the-art compilers like Ansor.
Deep learning models rely on highly optimized tensor libraries for efficient inference on heterogeneous hardware. Current deep compilers typically predetermine layouts of tensors and then optimize loops of operators. However, such unidirectional and one-off workflow strictly separates graph-level optimization and operator-level optimization into different system layers, missing opportunities for unified tuning. This paper proposes ALT, a compiler that performs joint graph- and operator-level optimizations for deep models. ALT provides a generic transformation module to manipulate layouts and loops with easy-to-use primitive functions. ALT further integrates an auto-tuning module that jointly optimizes graph-level data layouts and operator-level loops while guaranteeing efficiency. Experimental results show that ALT significantly outperforms state-of-the-art compilers (e.g., Ansor) in terms of both single operator performance (e.g., 1.5x speedup on average) and end-to-end inference performance (e.g., 1.4x speedup on average).