Learning Compressed Transforms with Low Displacement Rank
This addresses the problem of parameter efficiency in deep learning for practitioners needing compressed models, though it is an incremental improvement over existing LDR frameworks.
The paper tackles the problem of compressing neural network weight matrices by introducing a class of low displacement rank (LDR) matrices with learnable displacement operators and low-rank components, generalizing previous fixed-operator approaches. The result shows that this compact parameterization reduces sample complexity and achieves higher accuracy than existing compression methods, sometimes outperforming unstructured layers while using over 20x fewer parameters in image classification and language modeling tasks.
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR matrices with more general displacement operators, and explicitly learn over both the operators and the low-rank component. This class generalizes several previous constructions while preserving compression and efficient computation. We prove bounds on the VC dimension of multi-layer neural networks with structured weight matrices and show empirically that our compact parameterization can reduce the sample complexity of learning. When replacing weight layers in fully-connected, convolutional, and recurrent neural networks for image classification and language modeling tasks, our new classes exceed the accuracy of existing compression approaches, and on some tasks also outperform general unstructured layers while using more than 20x fewer parameters.