LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs
This work addresses noise robustness in CNNs for image classification, offering an incremental improvement over existing activation functions.
The authors tackled the problem of noise robustness in CNNs by proposing LayerAct, a novel activation mechanism that reduces the influence of input shifts without the limitations of Layer Normalization. Experimental results on three benchmark datasets show that LayerAct outperforms other activation functions on noisy datasets and often achieves superior performance on clean datasets.
In this work, we propose a novel activation mechanism called LayerAct for CNNs. This approach is motivated by our theoretical and experimental analyses, which demonstrate that Layer Normalization (LN) can mitigate a limitation of existing activation functions regarding noise robustness. However, LN is known to be disadvantageous in CNNs due to its tendency to make activation outputs homogeneous. The proposed method is designed to be more robust than existing activation functions by reducing the upper bound of influence caused by input shifts without inheriting LN's limitation. We provide analyses and experiments showing that LayerAct functions exhibit superior robustness compared to ElementAct functions. Experimental results on three clean and noisy benchmark datasets for image classification tasks indicate that LayerAct functions outperform other activation functions in handling noisy datasets while achieving superior performance on clean datasets in most cases.