Provable Target Sample Complexity Improvements as Pre-Trained Models Scale
This work addresses a foundational gap in understanding pre-trained model scaling for researchers and practitioners, though it is incremental as it builds on existing empirical observations.
The paper tackles the lack of theoretical explanation for why larger pre-trained models reduce downstream sample complexity, and it provides a novel framework that proves this improvement, offering justification for empirical scaling laws.
Pre-trained models have become indispensable for efficiently building models across a broad spectrum of downstream tasks. The advantages of pre-trained models have been highlighted by empirical studies on scaling laws, which demonstrate that larger pre-trained models can significantly reduce the sample complexity of downstream learning. However, existing theoretical investigations of pre-trained models lack the capability to explain this phenomenon. In this paper, we provide a theoretical investigation by introducing a novel framework, caulking, inspired by parameter-efficient fine-tuning (PEFT) methods such as adapter-based fine-tuning, low-rank adaptation, and partial fine-tuning. Our analysis establishes that improved pre-trained models provably decrease the sample complexity of downstream tasks, thereby offering theoretical justification for the empirically observed scaling laws relating pre-trained model size to downstream performance, a relationship not covered by existing results.