On consequences of finetuning on data with highly discriminative features
This addresses a problem for practitioners using transfer learning, as it highlights a potential performance degradation issue, but it appears incremental as it builds on known transfer learning drawbacks.
The paper identifies a drawback in transfer learning called 'feature erosion', where networks prioritize basic data patterns over valuable pre-learned features, analyzing its impact on performance and internal representations.
In the era of transfer learning, training neural networks from scratch is becoming obsolete. Transfer learning leverages prior knowledge for new tasks, conserving computational resources. While its advantages are well-documented, we uncover a notable drawback: networks tend to prioritize basic data patterns, forsaking valuable pre-learned features. We term this behavior "feature erosion" and analyze its impact on network performance and internal representations.