Making Sigmoid-MSE Great Again: Output Reset Challenges Softmax Cross-Entropy in Neural Network Classification
This work addresses classification tasks in neural networks, offering an incremental alternative to standard methods for improved robustness against noise.
The study tackled the problem of neural network classification by comparing Mean Squared Error (MSE) with sigmoid activation to the conventional Softmax Cross-Entropy (SCE), introducing the Output Reset algorithm to reduce errors and enhance robustness. It demonstrated that MSE achieves comparable accuracy and convergence rates to SCE on benchmark datasets like MNIST, CIFAR-10, and Fashion-MNIST, with superior performance in noisy data scenarios.
This study presents a comparative analysis of two objective functions, Mean Squared Error (MSE) and Softmax Cross-Entropy (SCE) for neural network classification tasks. While SCE combined with softmax activation is the conventional choice for transforming network outputs into class probabilities, we explore an alternative approach using MSE with sigmoid activation. We introduce the Output Reset algorithm, which reduces inconsistent errors and enhances classifier robustness. Through extensive experiments on benchmark datasets (MNIST, CIFAR-10, and Fashion-MNIST), we demonstrate that MSE with sigmoid activation achieves comparable accuracy and convergence rates to SCE, while exhibiting superior performance in scenarios with noisy data. Our findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.