Understanding Locally Competitive Networks
This work offers insights into improving neural network efficiency and interpretability, which is incremental but useful for researchers and practitioners in deep learning.
The paper investigates the self-modularization in neural networks with local competition activation functions, providing a unified explanation for their beneficial properties and demonstrating applications in efficient retrieval over large datasets.
Recently proposed neural network activation functions such as rectified linear, maxout, and local winner-take-all have allowed for faster and more effective training of deep neural architectures on large and complex datasets. The common trait among these functions is that they implement local competition between small groups of computational units within a layer, so that only part of the network is activated for any given input pattern. In this paper, we attempt to visualize and understand this self-modularization, and suggest a unified explanation for the beneficial properties of such networks. We also show how our insights can be directly useful for efficiently performing retrieval over large datasets using neural networks.