Rethinking Transfer Learning for Medical Image Classification
This work addresses the problem of inefficient and suboptimal transfer learning for medical image classification, offering a domain-specific incremental improvement over existing differential strategies.
The paper tackles the suboptimal performance of uniformly finetuning all layers in transfer learning for medical image classification by proposing TruncatedTL, a method that reuses and finetunes only appropriate bottom layers and discards the rest, resulting in superior performance and more compact models for efficient inference.
Transfer learning (TL) from pretrained deep models is a standard practice in modern medical image classification (MIC). However, what levels of features to be reused are problem-dependent, and uniformly finetuning all layers of pretrained models may be suboptimal. This insight has partly motivated the recent differential TL strategies, such as TransFusion (TF) and layer-wise finetuning (LWFT), which treat the layers in the pretrained models differentially. In this paper, we add one more strategy into this family, called TruncatedTL, which reuses and finetunes appropriate bottom layers and directly discards the remaining layers. This yields not only superior MIC performance but also compact models for efficient inference, compared to other differential TL methods. Our code is available at: https://github.com/sun-umn/TTL