Uniform Sampling from High-dimensional Spectral Norm Balls
Provides theoretical justification and a practical sampling method for high-dimensional spectral norm balls, relevant to optimization in large language models.
The paper proves that singular values of matrices sampled uniformly from the unit spectral norm ball converge to 1 almost surely as dimensions increase, enabling a simple sampling method for large matrices used in modern LLMs.
Motivated by an application in machine learning optimization, this paper focuses on the challenges of sampling a matrix uniformly from the unit spectral norm ball. It is proven that all singular values of sampled matrices converge to 1 almost surely as the matrix dimensions increase. This result provides the theoretical justification for a proposed simple sampling method applicable for large dimension sizes matching matrices found in modern large language models. Experimental results demonstrate both the convergence of the singular values, as well as the exact and proposed approximate sampling methods.