These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretraining
This addresses a fundamental limitation in transfer learning for machine learning practitioners, showing that large-scale pretraining may not be as effective as task-specific training when available.
The paper identifies an 'information saturation bottleneck' in deep learning models where networks fail to learn new features once they encode similar competing features during pretraining, leading to inconsistent performance on data distributions even within the training mixture. Empirical evidence suggests this phenomenon is pervasive, and the study proposes richer feature representations as a potential solution.
Transfer learning is a cornerstone of modern machine learning, promising a way to adapt models pretrained on a broad mix of data to new tasks with minimal new data. However, a significant challenge remains in ensuring that transferred features are sufficient to handle unseen datasets, amplified by the difficulty of quantifying whether two tasks are "related". To address these challenges, we evaluate model transfer from a pretraining mixture to each of its component tasks, assessing whether pretrained features can match the performance of task-specific direct training. We identify a fundamental limitation in deep learning models -- an "information saturation bottleneck" -- where networks fail to learn new features once they encode similar competing features during training. When restricted to learning only a subset of key features during pretraining, models will permanently lose critical features for transfer and perform inconsistently on data distributions, even components of the training mixture. Empirical evidence from published studies suggests that this phenomenon is pervasive in deep learning architectures -- factors such as data distribution or ordering affect the features that current representation learning methods can learn over time. This study suggests that relying solely on large-scale networks may not be as effective as focusing on task-specific training, when available. We propose richer feature representations as a potential solution to better generalize across new datasets and, specifically, present existing methods alongside a novel approach, the initial steps towards addressing this challenge.