Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations
This addresses a common issue in deep learning for label-insufficient situations, offering an incremental improvement over existing methods like entropy minimization.
The paper tackles the problem of reduced prediction diversity in label-insufficient learning scenarios by proposing Batch Nuclear-norm Maximization (BNM), which improves both discriminability and diversity in the output matrix, leading to performance gains in semi-supervised learning, domain adaptation, and open domain recognition.
The learning of the deep networks largely relies on the data with human-annotated labels. In some label insufficient situations, the performance degrades on the decision boundary with high data density. A common solution is to directly minimize the Shannon Entropy, but the side effect caused by entropy minimization, i.e., reduction of the prediction diversity, is mostly ignored. To address this issue, we reinvestigate the structure of classification output matrix of a randomly selected data batch. We find by theoretical analysis that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and rank of the batch output matrix. Besides, the nuclear-norm is an upperbound of the Frobenius-norm, and a convex approximation of the matrix rank. Accordingly, to improve both discriminability and diversity, we propose Batch Nuclear-norm Maximization (BNM) on the output matrix. BNM could boost the learning under typical label insufficient learning scenarios, such as semi-supervised learning, domain adaptation and open domain recognition. On these tasks, extensive experimental results show that BNM outperforms competitors and works well with existing well-known methods. The code is available at https://github.com/cuishuhao/BNM.