Language Model Self-improvement by Reinforcement Learning Contemplation
This addresses the need for cost-effective and scalable improvement of LLMs across various NLP tasks, though it is incremental as it builds on existing reinforcement learning and self-assessment concepts.
The paper tackles the problem of fine-tuning large language models (LLMs) without external supervision by introducing SIRLC, an unsupervised method where LLMs act as both student and teacher to self-improve through reinforcement learning, resulting in a 5.6% accuracy increase for reasoning tasks and a BERTScore rise from 0.82 to 0.86 for translation tasks.
Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing (NLP) tasks. However, fine-tuning these models often necessitates substantial supervision, which can be expensive and time-consuming to obtain. This paper introduces a novel unsupervised method called LanguageModel Self-Improvement by Reinforcement Learning Contemplation (SIRLC) that improves LLMs without reliance on external labels. Our approach is grounded in the observation that it is simpler for language models to assess text quality than to generate text. Building on this insight, SIRLC assigns LLMs dual roles as both student and teacher. As a student, the LLM generates answers to unlabeled questions, while as a teacher, it evaluates the generated text and assigns scores accordingly. The model parameters are updated using reinforcement learning to maximize the evaluation score. We demonstrate that SIRLC can be applied to various NLP tasks, such as reasoning problems, text generation, and machine translation. Our experiments show that SIRLC effectively improves LLM performance without external supervision, resulting in a 5.6% increase in answering accuracy for reasoning tasks and a rise in BERTScore from 0.82 to 0.86 for translation tasks. Furthermore, SIRLC can be applied to models of different sizes, showcasing its broad applicability.