Boosting Active Learning with Knowledge Transfer
This work addresses the problem of improving Active Learning for researchers and practitioners in computational biology and computer vision, though it appears incremental as it builds on existing teacher-student frameworks.
The paper tackles the challenge of uncertainty estimation in Active Learning, particularly for domain-specific tasks like cryo-ET classification, by proposing a knowledge transfer method using a teacher-student model to measure uncertainty without complex auxiliary designs, resulting in demonstrated efficacy and efficiency on computer vision and cryo-ET benchmarks.
Uncertainty estimation is at the core of Active Learning (AL). Most existing methods resort to complex auxiliary models and advanced training fashions to estimate uncertainty for unlabeled data. These models need special design and hence are difficult to train especially for domain tasks, such as Cryo-Electron Tomography (cryo-ET) classification in computational biology. To address this challenge, we propose a novel method using knowledge transfer to boost uncertainty estimation in AL. Specifically, we exploit the teacher-student mode where the teacher is the task model in AL and the student is an auxiliary model that learns from the teacher. We train the two models simultaneously in each AL cycle and adopt a certain distance between the model outputs to measure uncertainty for unlabeled data. The student model is task-agnostic and does not rely on special training fashions (e.g. adversarial), making our method suitable for various tasks. More importantly, we demonstrate that data uncertainty is not tied to concrete value of task loss but closely related to the upper-bound of task loss. We conduct extensive experiments to validate the proposed method on classical computer vision tasks and cryo-ET challenges. The results demonstrate its efficacy and efficiency.