Self-Knowledge Distillation for Learning Ambiguity
This addresses the issue of over-confidence in ambiguous natural language understanding tasks for language model users, representing an incremental improvement.
The paper tackles the problem of language models being over-confident on ambiguous samples by proposing a self-knowledge distillation method that learns better label distributions and re-calibrates confidence for highly ambiguous samples, resulting in significantly alleviated over-confidence and more efficient training compared to existing state-of-the-art methods.
Recent language models have shown remarkable performance on natural language understanding (NLU) tasks. However, they are often sub-optimal when faced with ambiguous samples that can be interpreted in multiple ways, over-confidently predicting a single label without consideration for its correctness. To address this issue, we propose a novel self-knowledge distillation method that enables models to learn label distributions more accurately by leveraging knowledge distilled from their lower layers. This approach also includes a learning phase that re-calibrates the unnecessarily strengthened confidence for training samples judged as extremely ambiguous based on the distilled distribution knowledge. We validate our method on diverse NLU benchmark datasets and the experimental results demonstrate its effectiveness in producing better label distributions. Particularly, through the process of re-calibrating the confidence for highly ambiguous samples, the issue of over-confidence when predictions for unseen samples do not match with their ground-truth labels has been significantly alleviated. This has been shown to contribute to generating better distributions than the existing state-of-the-art method. Moreover, our method is more efficient in training the models compared to the existing method, as it does not involve additional training processes to refine label distributions.