Decoupled Entropy Minimization
This addresses issues in imperfectly supervised learning tasks, such as noisy and dynamic environments, by improving EM to reduce class overlap and uncertainty, though it appears incremental as it builds on existing EM methods.
The paper tackled the limitations of classical Entropy Minimization (EM) by reformulating it into two components with opposite effects and proposing Adaptive Decoupled Entropy Minimization (AdaDEM), which outperforms an upper-bound variant and achieves superior performance in noisy and dynamic environments.
Entropy Minimization (EM) is beneficial to reducing class overlap, bridging domain gap, and restricting uncertainty for various tasks in machine learning, yet its potential is limited. To study the internal mechanism of EM, we reformulate and decouple the classical EM into two parts with opposite effects: cluster aggregation driving factor (CADF) rewards dominant classes and prompts a peaked output distribution, while gradient mitigation calibrator (GMC) penalizes high-confidence classes based on predicted probabilities. Furthermore, we reveal the limitations of classical EM caused by its coupled formulation: 1) reward collapse impedes the contribution of high-certainty samples in the learning process, and 2) easy-class bias induces misalignment between output distribution and label distribution. To address these issues, we propose Adaptive Decoupled Entropy Minimization (AdaDEM), which normalizes the reward brought from CADF and employs a marginal entropy calibrator (MEC) to replace GMC. AdaDEM outperforms DEM*, an upper-bound variant of classical EM, and achieves superior performance across various imperfectly supervised learning tasks in noisy and dynamic environments.