Regularized Deep Linear Discriminant Analysis
This work addresses an incremental improvement in representation learning for classification tasks, particularly in medical imaging and standard benchmarks.
The paper tackled the problem of Deep Linear Discriminant Analysis (DLDA) focusing too much on cooperative discriminative ability across dimensions and not enough on individual dimension separability, by proposing a regularization method on the within-class scatter matrix, which improved performance over DLDA and conventional networks on datasets like STL-10, CIFAR-10, and a pediatric chest X-ray dataset.
As a non-linear extension of the classic Linear Discriminant Analysis(LDA), Deep Linear Discriminant Analysis(DLDA) replaces the original Categorical Cross Entropy(CCE) loss function with eigenvalue-based loss function to make a deep neural network(DNN) able to learn linearly separable hidden representations. In this paper, we first point out DLDA focuses on training the cooperative discriminative ability of all the dimensions in the latent subspace, while put less emphasis on training the separable capacity of single dimension. To improve DLDA, a regularization method on within-class scatter matrix is proposed to strengthen the discriminative ability of each dimension, and also keep them complement each other. Experiment results on STL-10, CIFAR-10 and Pediatric Pneumonic Chest X-ray Dataset showed that our proposed regularization method Regularized Deep Linear Discriminant Analysis(RDLDA) outperformed DLDA and conventional neural network with CCE as objective. To further improve the discriminative ability of RDLDA in the local space, an algorithm named Subclass RDLDA is also proposed.