Attention, Distillation, and Tabularization: Towards Practical Neural Network-Based Prefetching
This work addresses the computational inefficiency of neural network-based prefetchers for computer architecture, enabling practical deployment with significant performance gains over existing methods.
The paper tackles the high inference latency of attention-based neural networks for memory access prediction in data prefetching by proposing a tabularization approach that converts matrix multiplications into fast table lookups, resulting in DART, which reduces arithmetic operations by 99.99% and accelerates inference by 170x while maintaining accuracy with only a 0.09 drop in F1-score.
Attention-based Neural Networks (NN) have demonstrated their effectiveness in accurate memory access prediction, an essential step in data prefetching. However, the substantial computational overheads associated with these models result in high inference latency, limiting their feasibility as practical prefetchers. To close the gap, we propose a new approach based on tabularization that significantly reduces model complexity and inference latency without sacrificing prediction accuracy. Our novel tabularization methodology takes as input a distilled, yet highly accurate attention-based model for memory access prediction and efficiently converts its expensive matrix multiplications into a hierarchy of fast table lookups. As an exemplar of the above approach, we develop DART, a prefetcher comprised of a simple hierarchy of tables. With a modest 0.09 drop in F1-score, DART reduces 99.99% of arithmetic operations from the large attention-based model and 91.83% from the distilled model. DART accelerates the large model inference by 170x and the distilled model by 9.4x. DART has comparable latency and storage costs as state-of-the-art rule-based prefetcher BO but surpasses it by 6.1% in IPC improvement. DART outperforms state-of-the-art NN-based prefetchers TransFetch by 33.1% and Voyager by 37.2% in terms of IPC improvement, primarily due to its low prefetching latency.