FTuner: A Fast Dynamic Shape Tensors Program Auto-Tuner for Deep Learning Compilers
This work addresses performance optimization for dynamic tensor shapes in deep learning compilers, offering a significant speed improvement and drastically reduced tuning time, though it is incremental as it builds on existing auto-tuning approaches.
The paper tackles the challenge of optimizing deep learning models with dynamic tensor shapes by introducing FTuner, a fast auto-tuner that uses uKernels and an analytic hardware model to match input shapes, achieving a 3% speedup and reducing tuning time by two orders of magnitude compared to existing methods.
Many artificial intelligence models process input data of different lengths and resolutions, making the shape of the tensors dynamic. The performance of these models depends on the shape of the tensors, which makes it difficult to optimize the tensors before the model runs. There are two common solutions to this problem. The first is to add useless data to the input to match a pre-optimized tensor library. The second is to use small basic tensors to create a tensor that is closest in size to the input data and then tune it to minimize padding. However, this second solution can be time-consuming. This paper proposes a new technique for deep learning compilers called FTuner. Instead of using a large design space or training a cost model, we use an abstract computational unit called the uKernel to patch together small, various-sized tensors to match the shape of the input tensor. We determine the shape of the uKernel using an analytic hardware information model. Experiments show that the FTuner can achieve comparable operators and end-to-end performance to vendor libraries and achieves 3\% speedup on existing auto-tuner with the model-training compiler while reducing tuning time by two orders of magnitude.