High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
This work addresses the inefficiency of training large models for deployment, offering a method to reduce model size without significant loss, which is incremental but practical for AI practitioners.
The paper tackles the problem of model redundancy in high-dimensional optimization by showing that representations of models like ResNet, ViT, and BERT can be compressed up to 16x with only about 1% performance degradation, using data-independent random projections that match PCA and learned methods.
State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution manifold is intrinsically robust. These findings establish the foundation for Subspace-Native Distillation: a paradigm where student models target this intrinsic manifold directly, bypassing the high-dimensional optimization bottleneck to realize the vision of "Train Big, Deploy Small"