Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
This highlights challenges in achieving consistent ML behavior across diverse hardware, which is critical for developers and users relying on cost-effective alternatives in cloud data centers.
The paper conducted the first empirical study on machine learning model behavior across heterogeneous AI accelerators, revealing that newer platforms from Mac and Huawei support at least 17% fewer operators and exhibit over 5% output discrepancies compared to NVIDIA.
While NVIDIA remains the dominant provider of AI accelerators within cloud data center, emerging vendors such as AMD, Intel, Mac, and Huawei offer cost-effective alternatives with claims of compatibility and performance. This paper presents the first empirical study investigating divergence in machine learning model across heterogeneous AI accelerators. Utilizing an automated pipeline, we synthesize over 100,000 variant models derived from 4,000 real-world models and execute them across five different enterprise-grade accelerators. Our findings suggest that newer AI platforms from Mac and Huawei support at least 17\% fewer operators than NVIDIA. These platforms also exhibit a higher rate of output discrepancies (exceeding 5\%), which stem from differences in operator implementations, handling of exceptional numerical values, and instruction scheduling. They are also more susceptible to failures during model compilation-based acceleration, and in some cases, the compiled models produce outputs that differ noticeably from those generated using the standard execution mode. In addition, we identify 7 implementation flaws in PyTorch and 40 platform-specific issues across vendors. These results underscore the challenges of achieving consistent machine learning behavior in an increasingly diverse hardware ecosystem.