Theoretical Modeling of LLM Self-Improvement Training Dynamics Through Solver-Verifier Gap
This work addresses a theoretical gap in self-improvement techniques for LLMs, providing insights into training dynamics, but it is incremental as it builds on existing concepts without introducing a new method.
The paper tackles the problem of understanding how large language models (LLMs) evolve during self-improvement training by theoretically modeling the dynamics using a solver-verifier gap, and empirically validates this framework on various LLMs and datasets, showing it can quantify capability limits and analyze external data effects.
Self-improvement is among the most prominent techniques within the realm of large language models (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performances evolve during the self-improvement process remains underexplored. In this paper, we theoretically model the training dynamics of self-improvement via the concept of solver-verifier gap. This is inspired by the conjecture that the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We empirically validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond self-improvement, we extend our analysis to investigate how external data influences these dynamics within the framework. Notably, we find that under limited external data regimes, such external data can be utilized at any stage without significantly affecting final performances, which accords with the empirical observations.