Sparsely Activated Networks: A new method for decomposing and compressing data
This work addresses the need for more efficient and interpretable unsupervised learning representations, though it appears incremental as it builds on existing sparse activation methods.
The authors tackled the problem of unsupervised learning models not considering representation compression by introducing the φ metric to evaluate models based on reconstruction accuracy and compression, and proposed Sparsely Activated Networks (SANs) with sparse activation functions, showing that models selected using φ achieve small description length and interpretable kernels on datasets like Physionet and MNIST.
Recent literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features, but without considering the description length of the representations. In this thesis, first we introduce the $\varphi$ metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined metric $\varphi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\varphi$ have small description representation length and consist of interpretable kernels.