Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfitting
This work addresses a gap in understanding training dynamics and error bounds for transformers, which is incremental as it builds on existing concepts of overfitting.
The paper tackles the problem of understanding generalization in transformers by developing a generalization theory for a two-layer transformer with labeled flip noise, presenting error bounds for benign and harmful overfitting under varying signal-to-noise ratios and categorizing training dynamics into three stages with corresponding bounds.
Transformers serve as the foundational architecture for many successful large-scale models, demonstrating the ability to overfit the training data while maintaining strong generalization on unseen data, a phenomenon known as benign overfitting. However, research on how the training dynamics influence error bounds within the context of benign overfitting has been limited. This paper addresses this gap by developing a generalization theory for a two-layer transformer with labeled flip noise. Specifically, we present generalization error bounds for both benign and harmful overfitting under varying signal-to-noise ratios (SNR), where the training dynamics are categorized into three distinct stages, each with its corresponding error bounds. Additionally, we conduct extensive experiments to identify key factors that influence test errors in transformers. Our experimental results align closely with the theoretical predictions, validating our findings.