From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference
This work provides a theoretical foundation for a broader class of nonlinearities in deep learning, benefiting researchers and practitioners by explaining and improving activation functions like swish, though it is incremental in extending existing MASO theory.
The paper tackles the problem of understanding deep network nonlinearities by extending the max-affine spline operator (MASO) framework to include non-piecewise affine and convex activation functions like sigmoid and softmax, linking them to probabilistic Gaussian Mixture Models and vector quantization, and shows that enforcing orthogonality in linear filters improves performance with significant gains in experiments.
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling. In particular, DN layers constructed from these operations can be interpreted as {\em max-affine spline operators} (MASOs) that have an elegant link to vector quantization (VQ) and $K$-means. While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax. {\em This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).} We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural "hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding "soft" VQ inference problems. We further extend the framework by hybridizing the hard and soft VQ optimizations to create a $β$-VQ inference that interpolates between hard, soft, and linear VQ inference. A prime example of a $β$-VQ DN nonlinearity is the {\em swish} nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation. Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.