Generalization vs. Memorization in the Presence of Statistical Biases in Transformers
This addresses the problem of overestimating AI generalization for researchers, but it is incremental as it builds on prior work on spurious correlations.
The study investigated how statistical biases affect transformer models' generalization on algorithmic tasks, finding that these biases impair out-of-distribution performance and lead to overestimation of generalization capabilities.
This study aims to understand how statistical biases affect the model's ability to generalize to in-distribution and out-of-distribution data on algorithmic tasks. Prior research indicates that transformers may inadvertently learn to rely on these spurious correlations, leading to an overestimation of their generalization capabilities. To investigate this, we evaluate transformer models on several synthetic algorithmic tasks, systematically introducing and varying the presence of these biases. We also analyze how different components of the transformer models impact their generalization. Our findings suggest that statistical biases impair the model's performance on out-of-distribution data, providing a overestimation of its generalization capabilities. The models rely heavily on these spurious correlations for inference, as indicated by their performance on tasks including such biases.