Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks
This work addresses model compression for resource-constrained environments, providing theoretical insights that are incremental but foundational for algorithm development.
The paper tackles the problem of high memory and computational demands in deep neural networks by developing an analytical framework for data-driven post-training low-rank compression, proving recovery theorems under varying assumptions about activation structure to explain performance advantages over data-agnostic methods.
Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environments. Model compression techniques, such as low-rank approximation, offer a promising solution by reducing the size and complexity of these networks while only minimally sacrificing accuracy. In this paper, we develop an analytical framework for data-driven post-training low-rank compression. We prove three recovery theorems under progressively weaker assumptions about the approximate low-rank structure of activations, modeling deviations via noise. Our results represent a step toward explaining why data-driven low-rank compression methods outperform data-agnostic approaches and towards theoretically grounded compression algorithms that reduce inference costs while maintaining performance.