Victor Pan

Liang Zhao, Siyu Liao, Yanzhi Wang et al.

Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

Victor Pan, John Svadlenka, Liang Zhao

Low-rank approximation of a matrix by means of structured random sampling has been consistently efficient in its extensive empirical studies around the globe, but adequate formal support for this empirical phenomenon has been missing so far. Based on our novel insight into the subject, we provide such an elusive formal support and derandomize and simplify the known numerical algorithms for low-rank approximation and related computations. Our techniques can be applied to some other areas of fundamental matrix computations, in particular to the Least Squares Regression, Gaussian elimination with no pivoting and block Gaussian elimination. Our formal results and our numerical tests are in good accordance with each other.