Transfer-Tuning: Reusing Auto-Schedules for Efficient Tensor Program Code Generation
This addresses the problem of inefficient auto-scheduling for deep neural networks, particularly benefiting developers and researchers by reducing tuning time, though it is incremental as it builds on existing auto-scheduling methods.
The paper tackles the time-consuming process of auto-scheduling for tensor programs by introducing transfer-tuning, which reuses auto-schedules from pre-tuned models to reduce inference time. It achieves up to 88.41% (49.13% on average) of the maximum speedup with 6.5x less search time compared to the state-of-the-art Ansor auto-scheduler on server-class CPUs.
Auto-scheduling for tensor programs is a process where a search algorithm automatically explores candidate schedules (program transformations) for a given program on a target hardware platform to improve its performance. However this can be a very time consuming process depending on the complexity of the tensor program and the capacity of the target device, with often many thousands of program variants being explored. To address this, in this paper we introduce the idea of transfer-tuning, a novel approach to identify and reuse auto-schedules between tensor programs. We demonstrate this concept using Deep Neural Networks (DNNs), taking sets of auto-schedules from pre-tuned DNNs and using them to reduce the inference time of a new DNN. We compare transfer-tuning against the state-of-the-art Ansor auto-scheduler, defining the maximum possible speedup for a given DNN model as what Ansor achieves using its recommended full tuning time. On a server-class CPU and across 11 widely used DNN models, we observe that transfer-tuning achieves up to $88.41\%$ ($49.13\%$ on average) of this maximum speedup, while Ansor requires $6.5\times$ more search time on average to match it. We also evaluate transfer-tuning on a constrained edge CPU and observe that the differences in search time are exacerbated, with Ansor requiring $10.8\times$ more time on average to match transfer-tuning's speedup, which further demonstrates its value. Our code is available at https://www.github.com/gicLAB/transfer-tuning