Self-Questioning Language Models
This addresses the challenge of data scarcity for AI researchers by enabling self-improvement in language models, though it is incremental as it builds on existing self-play and reinforcement learning techniques.
The paper tackles the problem of improving large language models' reasoning skills without external data by proposing Self-Questioning Language Models (SQLM), an asymmetric self-play framework where models generate and solve their own questions, resulting in performance gains on benchmarks like three-digit multiplication, algebra problems, and programming tasks.
Can large language models improve without external data -- by generating their own questions and answers? We hypothesize that a pre-trained language model can improve its reasoning skills given only a single prompt specifying the topic (e.g., algebra word problems) and asking the model to generate its own questions. To do this, we propose Self-Questioning Language Models (SQLM): an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver, who tries to answer it. Both the proposer and solver are trained via reinforcement learning. The proposer receives a reward if the problem is not too easy or too difficult, and the solver receives a reward based on majority voting, a proxy for correctness in the absence of ground-truth answers. For coding, the proposer can instead generate unit tests which are used for verification. We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces. By continually generating more interesting problems and attempting to solve them, language models can improve on downstream benchmarks without access to any curated training datasets.