Learning sparse transformations through backpropagation
This work addresses the challenge of efficiently learning sparse structures in deep learning models, which is important for researchers and practitioners in AI to reduce computational costs and improve interpretability, though it is incremental as it builds on existing backpropagation and sparse representation techniques.
The authors tackled the problem of learning sparse transformations in deep neural networks by introducing the adaptive sparse hyperlayer, which parameterizes sparsity directly through index-tuples and values, and they demonstrated its effectiveness by achieving competitive performance on real-world tasks, such as visual classification and differentiable sorting on MNIST digits.
Many transformations in deep learning architectures are sparsely connected. When such transformations cannot be designed by hand, they can be learned, even through plain backpropagation, for instance in attention mechanisms. However, during learning, such sparse structures are often represented in a dense form, as we do not know beforehand which elements will eventually become non-zero. We introduce the adaptive, sparse hyperlayer, a method for learning a sparse transformation, paramatrized sparsely: as index-tuples with associated values. To overcome the lack of gradients from such a discrete structure, we introduce a method of randomly sampling connections, and backpropagating over the randomly wired computation graph. To show that this approach allows us to train a model to competitive performance on real data, we use it to build two architectures. First, an attention mechanism for visual classification. Second, we implement a method for differentiable sorting: specifically, learning to sort unlabeled MNIST digits, given only the correct order.