Exploring connections of spectral analysis and transfer learning in medical imaging
This work addresses shortcut learning in medical imaging for AI practitioners, offering insights into transfer learning but is incremental as it builds on existing spectral analysis methods.
The paper investigated how spectral analysis reveals model sensitivity to frequency shortcuts in medical imaging transfer learning, finding that alignment between a model's learning priority and artifact power spectrum leads to overfitting, and demonstrated that editing source data can alter resistance to shortcut learning.
In this paper, we use spectral analysis to investigate transfer learning and study model sensitivity to frequency shortcuts in medical imaging. By analyzing the power spectrum density of both pre-trained and fine-tuned model gradients, as well as artificially generated frequency shortcuts, we observe notable differences in learning priorities between models pre-trained on natural vs medical images, which generally persist during fine-tuning. We find that when a model's learning priority aligns with the power spectrum density of an artifact, it results in overfitting to that artifact. Based on these observations, we show that source data editing can alter the model's resistance to shortcut learning.