Utilizing Static Analysis and Code Generation to Accelerate Neural Networks
This addresses the need for easier and more efficient neural network development, particularly for researchers without expert systems knowledge, by providing a domain-specific compiler that reduces code complexity and accelerates performance.
The paper tackles the problem of neural network applications being limited by computational power and development complexity, presenting SONNC, a compiler that uses static analysis to generate optimized parallel code, resulting in performance improvements of up to 7.8x over hand-optimized C++ and 24x over MATLAB.
As datasets continue to grow, neural network (NN) applications are becoming increasingly limited by both the amount of available computational power and the ease of developing high-performance applications. Researchers often must have expert systems knowledge to make their algorithms run efficiently. Although available computing power increases rapidly each year, algorithm efficiency is not able to keep pace due to the use of general purpose compilers, which are not able to fully optimize specialized application domains. Within the domain of NNs, we have the added knowledge that network architecture remains constant during training, meaning the architecture's data structure can be statically optimized by a compiler. In this paper, we present SONNC, a compiler for NNs that utilizes static analysis to generate optimized parallel code. We show that SONNC's use of static optimizations make it able to outperform hand-optimized C++ code by up to 7.8X, and MATLAB code by up to 24X. Additionally, we show that use of SONNC significantly reduces code complexity when using structurally sparse networks.