Deep Recommender Models Inference: Automatic Asymmetric Data Flow Optimization
This addresses a critical efficiency problem for companies like Meta, where DLRMs account for over 79% of AI workloads, though it is incremental as it optimizes existing methods for a known bottleneck.
The paper tackled the performance bottleneck in Deep Recommender Models (DLRMs) inference, specifically in embedding layers, by proposing tailored data flows and an automatic asymmetric mapping framework, achieving speed-ups of 1.5x to 6.5x for real workloads and over 20x for unbalanced distributions.
Deep Recommender Models (DLRMs) inference is a fundamental AI workload accounting for more than 79% of the total AI workload in Meta's data centers. DLRMs' performance bottleneck is found in the embedding layers, which perform many random memory accesses to retrieve small embedding vectors from tables of various sizes. We propose the design of tailored data flows to speedup embedding look-ups. Namely, we propose four strategies to look up an embedding table effectively on one core, and a framework to automatically map the tables asymmetrically to the multiple cores of a SoC. We assess the effectiveness of our method using the Huawei Ascend AI accelerators, comparing it with the default Ascend compiler, and we perform high-level comparisons with Nvidia A100. Results show a speed-up varying from 1.5x up to 6.5x for real workload distributions, and more than 20x for extremely unbalanced distributions. Furthermore, the method proves to be much more independent of the query distribution than the baseline.