Multi-Bias Non-linear Activation in Deep Neural Networks
This addresses the problem of redundant filters and computational inefficiency in deep neural networks for computer vision, offering a simple yet effective solution for improved feature representation.
The paper tackles the limitation of ReLU and its variants in separating noise and signal based solely on magnitude, proposing a multi-bias non-linear activation (MBA) layer to decouple responses into multiple maps by multi-thresholding magnitudes, achieving state-of-the-art performance on several benchmarks.
As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classification of noise and signal not only depends on the magnitude of responses, but also the context of how the feature responses would be used to detect more abstract patterns in higher layers. In order to output multiple response maps with magnitude in different ranges for a particular visual pattern, existing networks employing ReLU and its variants have to learn a large number of redundant filters. In this paper, we propose a multi-bias non-linear activation (MBA) layer to explore the information hidden in the magnitudes of responses. It is placed after the convolution layer to decouple the responses to a convolution kernel into multiple maps by multi-thresholding magnitudes, thus generating more patterns in the feature space at a low computational cost. It provides great flexibility of selecting responses to different visual patterns in different magnitude ranges to form rich representations in higher layers. Such a simple and yet effective scheme achieves the state-of-the-art performance on several benchmarks.