COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization
This addresses a fundamental training challenge in deep learning for researchers and practitioners, though it appears incremental as it builds on existing soft label methods.
The paper tackles the chicken-egg problem of needing a well-trained learner to generate soft labels for improving DNN training by proposing COLAM, a framework that co-learns DNNs and soft labels via alternating minimization, resulting in improved testing classification accuracy on many tasks.
Softening labels of training datasets with respect to data representations has been frequently used to improve the training of deep neural networks (DNNs). While such a practice has been studied as a way to leverage privileged information about the distribution of the data, a well-trained learner with soft classification outputs should be first obtained as a prior to generate such privileged information. To solve such chicken-egg problem, we propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives - (a) the training loss subject to soft labels and (b) the objective to learn improved soft labels - in one end-to-end training procedure. We performed extensive experiments to compare our proposed method with a series of baselines. The experiment results show that COLAM achieves improved performance on many tasks with better testing classification accuracy. We also provide both qualitative and quantitative analyses that explain why COLAM works well.