GmNet: Revisiting Gating Mechanisms From A Frequency View
This work addresses a theoretical gap for researchers in neural network design, though it appears incremental as it builds on existing gating concepts.
The paper tackled the lack of theoretical analysis on gating mechanisms in neural networks by exploring their effects from a frequency perspective, proposing GmNet which minimizes low-frequency bias and achieves impressive performance in image classification.
Gating mechanisms have emerged as an effective strategy integrated into model designs beyond recurrent neural networks for addressing long-range dependency problems. In a broad understanding, it provides adaptive control over the information flow while maintaining computational efficiency. However, there is a lack of theoretical analysis on how the gating mechanism works in neural networks. In this paper, inspired by the \textit{convolution theorem}, we systematically explore the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective. We investigate the interact between the element-wise product and activation functions in managing the responses to different frequency components. Leveraging these insights, we propose a Gating Mechanism Network (GmNet), a lightweight model designed to efficiently utilize the information of various frequency components. It minimizes the low-frequency bias present in existing lightweight models. GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task.