Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
This work addresses the problem of enhancing language model's reasoning capabilities, particularly for auto-regressive models, which is significant for natural language processing applications.
The authors tackled the challenge of generating long and correct Chain-of-Thought trajectories by proposing RELAY, which aligns CoT reasoning with loop iterations and applies intermediate supervision, resulting in significant improvements in auto-regressive model performance. The approach enables the model to predict CoT reasoning steps for unseen data and generate accurate reasoning chains for complex problems.
Chain-of-Thought (CoT) prompting has emerged as a powerful technique for enhancing language model's reasoning capabilities. However, generating long and correct CoT trajectories is challenging. Recent studies have demonstrated that Looped Transformers possess remarkable length generalization capabilities, but their limited generality and adaptability prevent them from serving as an alternative to auto-regressive solutions. To better leverage the strengths of Looped Transformers, we propose RELAY (REasoning through Loop Alignment iterativelY). Specifically, we align the steps of Chain-of-Thought (CoT) reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. This additional iteration-wise supervision not only preserves the Looped Transformer's ability for length generalization but also enables it to predict CoT reasoning steps for unseen data. Therefore, we leverage this Looped Transformer to generate accurate reasoning chains for complex problems that exceed the training length, which will then be used to fine-tune an auto-regressive model. We conduct extensive experiments, and the results demonstrate the effectiveness of our approach, with significant improvements in the performance of the auto-regressive model. Code will be released at https://github.com/qifanyu/RELAY.