Adjustable Bounded Rectifiers: Towards Deep Binary Representations
This work addresses the problem of deploying more sophisticated models in resource-constrained environments by saving memory space, though it is incremental in advancing binary representation techniques.
The paper tackles the challenge of learning binary representations in deep neural networks to improve memory efficiency and computation speed, achieving results where the last layer can be binarized without performance loss and all layers with small degradations on datasets like MNIST, CIFAR10, and ILSVRC2012.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we softly encourage activations to diverge from real values to binary by approximating step functions. Our final representation is completely binary. We test our approach on MNIST, CIFAR10, and ILSVRC2012 dataset, and systematically study the training dynamics of the binarization process. Our approach can binarize the last layer representation without loss of performance and binarize all the layers with reasonably small degradations. The memory space that it saves may allow more sophisticated models to be deployed, thus compensating the loss. To the best of our knowledge, this is the first work to report results on current deep network architectures using complete binary middle representations. Given the learned representations, we find that the firing or inhibition of a binary neuron is usually associated with a meaningful interpretation across different classes. This suggests that the semantic structure of a neural network may be manifested through a guided binarization process.