Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
This addresses the problem of improving generalization in large language models for tasks like arithmetic and maze solving, though it is incremental as it builds on existing transformer architectures without architectural changes.
The paper tackles the problem of large language models struggling with length generalization and solving complex problems beyond their training distribution by introducing a self-improvement approach where models iteratively generate and learn from their own solutions. The result shows that this method enables models to solve problems far beyond their initial training, such as generalizing from 10-digit to 100-digit addition without saturation, with exponential improvements in out-of-distribution performance in some cases.
Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach where models iteratively generate and learn from their own solutions, progressively tackling harder problems while maintaining a standard transformer architecture. Across diverse tasks including arithmetic, string manipulation, and maze solving, self-improving enables models to solve problems far beyond their initial training distribution-for instance, generalizing from 10-digit to 100-digit addition without apparent saturation. We observe that in some cases filtering for correct self-generated examples leads to exponential improvements in out-of-distribution performance across training rounds. Additionally, starting from pretrained models significantly accelerates this self-improvement process for several tasks. Our results demonstrate how controlled weak-to-strong curricula can systematically teach a model logical extrapolation without any changes to the positional embeddings, or the model architecture.