Improving Discrete Latent Representations With Differentiable Approximation Bridges
This method addresses a fundamental bottleneck in neural network training for researchers and practitioners by expanding the class of usable functions, though it is incremental as it builds on existing estimators.
The paper tackles the problem of using non-differentiable functions in neural networks by introducing a differentiable approximation bridge (DAB) to enable gradient updates, resulting in improved performance such as a 10% increase in image reconstruction quality and a 77% accuracy improvement in neural sequence sorting.
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present strong empirical results (performing over 600 experiments) in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator [37, 26] as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX [16] discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator [5] in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks.