Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices
This work addresses the problem of reducing computational and memory costs for deep learning models in speech processing tasks, offering an incremental improvement over existing compression methods.
The paper tackles the problem of compressing neural networks by replacing 1x1-convolutions in 1D time-channel separable convolutions with constant, sparse random ternary matrices, which eliminates multiplications and training requirements. This approach improves Pareto frontiers, achieving a state-of-the-art accuracy increase from 97.21% to 97.41% on Google Speech Commands v1 and reducing weights by half with only a 1% word error rate sacrifice on Librispeech.
We demonstrate that 1x1-convolutions in 1D time-channel separable convolutions may be replaced by constant, sparse random ternary matrices with weights in $\{-1,0,+1\}$. Such layers do not perform any multiplications and do not require training. Moreover, the matrices may be generated on the chip during computation and therefore do not require any memory access. With the same parameter budget, we can afford deeper and more expressive models, improving the Pareto frontiers of existing models on several tasks. For command recognition on Google Speech Commands v1, we improve the state-of-the-art accuracy from $97.21\%$ to $97.41\%$ at the same network size. Alternatively, we can lower the cost of existing models. For speech recognition on Librispeech, we half the number of weights to be trained while only sacrificing about $1\%$ of the floating-point baseline's word error rate.