Self-Knowledge Distillation via Dropout
This work addresses efficiency and performance issues in deep learning for vision applications, offering a simple, parameter-free method that is incremental but broadly applicable.
The paper tackles the computational and memory costs of deep neural networks by proposing a self-knowledge distillation method using dropout (SD-Dropout), which improves generalization across vision tasks like image classification and object detection, with demonstrated gains in calibration, robustness, and out-of-distribution detection.
To boost the performance, deep neural networks require deeper or wider network structures that involve massive computational and memory costs. To alleviate this issue, the self-knowledge distillation method regularizes the model by distilling the internal knowledge of the model itself. Conventional self-knowledge distillation methods require additional trainable parameters or are dependent on the data. In this paper, we propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout distills the posterior distributions of multiple models through a dropout sampling. Our method does not require any additional trainable modules, does not rely on data, and requires only simple operations. Furthermore, this simple method can be easily combined with various self-knowledge distillation approaches. We provide a theoretical and experimental analysis of the effect of forward and reverse KL-divergences in our work. Extensive experiments on various vision tasks, i.e., image classification, object detection, and distribution shift, demonstrate that the proposed method can effectively improve the generalization of a single network. Further experiments show that the proposed method also improves calibration performance, adversarial robustness, and out-of-distribution detection ability.