An Efficient Batch Solver for the Singular Value Decomposition on GPUs
This work addresses a bottleneck in high-performance computing for applications like PCA and low-rank approximations, though it is incremental in nature.
The paper tackled the problem of efficiently solving numerous small singular value decomposition (SVD) problems on GPUs, introducing a solver that achieved significant performance speedups over existing solutions on both NVIDIA and AMD systems.
The singular value decomposition (SVD) is a powerful tool in modern numerical linear algebra, which underpins computational methods such as principal component analysis (PCA), low-rank approximations, and randomized algorithms. Many practical scenarios require solving numerous small SVD problems, a regime generally referred to as "batch SVD". Existing programming models can handle this efficiently on parallel CPU architectures, but high-performance solutions for GPUs remain immature. A GPU-oriented batch SVD solver is introduced. This solver exploits the one-sided Jacobi algorithm to exploit fine-grained parallelism, and a number of algorithmic and design optimizations achieve unmatched performance. Starting from a baseline solver, a sequence of optimizations is applied to obtain incremental performance gains. Numerical experiments show that the new solver is robust across problems with different numerical properties, matrix shapes, and arithmetic precisions. Performance benchmarks on both NVIDIA and AMD systems show significant performance speedups over vendor solutions as well as existing open-source solvers.