Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
This work addresses computational efficiency for language model practitioners, though it appears incremental as it builds on existing depth-recurrent methods.
The paper tackled the problem of converting pretrained non-recurrent language models into depth-recurrent models to reduce computational cost while preserving performance, finding that this approach yields better performance at a given compute budget than post-training the original models, as demonstrated in mathematics experiments.
Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.