Fake Runs, Real Fixes -- Analyzing xPU Performance Through Simulation
This work addresses performance optimization for ML accelerators, which is crucial for efficiency as models grow, but it is incremental as it builds on existing simulation tools.
The paper tackled the problem of coarse-grained performance analysis for ML accelerators by introducing xPU-Shark, a fine-grained methodology using hardware-level simulation, which revealed inefficiencies and led to optimizations improving communication collective performance by up to 15% and reducing token generation latency by up to 4.1%.
As models become larger, ML accelerators are a scarce resource whose performance must be continually optimized to improve efficiency. Existing performance analysis tools are coarse grained, and fail to capture model performance at the machine-code level. In addition, these tools often do not provide specific recommendations for optimizations. We present xPU-Shark, a fine-grained methodology for analyzing ML models at the machine-code level that provides actionable optimization suggestions. Our core insight is to use a hardware-level simulator, an artifact of the hardware design process that we can re-purpose for performance analysis. xPU-Shark captures traces from production deployments running on accelerators and replays them in a modified microarchitecture simulator to gain low-level insights into the model's performance. We implement xPU-Shark for our in-house accelerator and used it to analyze the performance of several of our production LLMs, revealing several previously-unknown microarchitecture inefficiencies. Leveraging these insights, we optimize a common communication collective by up to 15% and reduce token generation latency by up to 4.1%.