Axiomatization of Gradient Smoothing in Neural Networks
This work provides a foundational theoretical framework for gradient smoothing in neural networks, which is incremental as it builds on existing methods to offer a more systematic approach.
The paper tackled the problem of noisy gradients in neural networks by proposing a theoretical framework based on function mollification and Monte Carlo integration to axiomatize gradient smoothing, and it demonstrated the framework's potential through experimental measurements of newly designed methods.
Gradients play a pivotal role in neural networks explanation. The inherent high dimensionality and structural complexity of neural networks result in the original gradients containing a significant amount of noise. While several approaches were proposed to reduce noise with smoothing, there is little discussion of the rationale behind smoothing gradients in neural networks. In this work, we proposed a gradient smooth theoretical framework for neural networks based on the function mollification and Monte Carlo integration. The framework intrinsically axiomatized gradient smoothing and reveals the rationale of existing methods. Furthermore, we provided an approach to design new smooth methods derived from the framework. By experimental measurement of several newly designed smooth methods, we demonstrated the research potential of our framework.