The Power of Certainty: How Confident Models Lead to Better Segmentation
This work addresses the challenge of deploying efficient and generalizable polyp segmentation models in clinical settings, though it appears incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of overfitting and poor generalization in large, over-parameterized deep learning models for polyp segmentation in colonoscopy by proposing a confidence-based self-distillation approach that outperforms state-of-the-art models and generalizes well across datasets from multiple clinical centers.
Deep learning models have been proposed for automatic polyp detection and precise segmentation of polyps during colonoscopy procedures. Although these state-of-the-art models achieve high performance, they often require a large number of parameters. Their complexity can make them prone to overfitting, particularly when trained on biased datasets, and can result in poor generalization across diverse datasets. Knowledge distillation and self-distillation are proposed as promising strategies to mitigate the limitations of large, over-parameterized models. These approaches, however, are resource-intensive, often requiring multiple models and significant memory during training. We propose a confidence-based self-distillation approach that outperforms state-of-the-art models by utilizing only previous iteration data storage during training, without requiring extra computation or memory usage during testing. Our approach calculates the loss between the previous and current iterations within a batch using a dynamic confidence coefficient. To evaluate the effectiveness of our approach, we conduct comprehensive experiments on the task of polyp segmentation. Our approach outperforms state-of-the-art models and generalizes well across datasets collected from multiple clinical centers. The code will be released to the public once the paper is accepted.