Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning
This work addresses the challenge of optimizing error measures in neural network training for researchers and practitioners, though it appears incremental by building on prior measures like $E_{Exp}$.
The paper tackles the problem of selecting error measures for neural network training by proposing a novel error measure, $E_{ExpAbs}$, that dynamically integrates statistical and information theory approaches, resulting in improved model accuracy and training process efficiency.
The paper analyses and serves with a positioning of various error measures applied in neural network training and identifies that there is no best of measure, although there is a set of measures with changing superiorities in different learning situations. An outstanding, remarkable measure called $E_{Exp}$ published by Silva and his research partners represents a research direction to combine more measures successfully with fixed importance weighting during learning. The main idea of the paper is to go far beyond and to integrate this relative importance into the neural network training algorithm(s) realized through a novel error measure called $E_{ExpAbs}$. This approach is included into the Levenberg-Marquardt training algorithm, so, a novel version of it is also introduced, resulting a self-adaptive, dynamic learning algorithm. This dynamism does not has positive effects on the resulted model accuracy only, but also on the training process itself. The described comprehensive algorithm tests proved that the proposed, novel algorithm integrates dynamically the two big worlds of statistics and information theory that is the key novelty of the paper.