Hankel Singular Value Regularization for Highly Compressible State Space Models
This work addresses the problem of model compression for practitioners using state space models in sequence tasks, offering a scalable method to reduce model size without sacrificing performance.
The paper tackles the challenge of compressing state space models in deep neural networks for long-range sequence tasks by introducing Hankel singular value regularization, which results in models that are up to 10 times more compressible while maintaining high accuracy.
Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy.