On the Convergence of Self-Improving Online LLM Alignment
For researchers working on online LLM alignment, this work provides a theoretical convergence guarantee and a practical improvement over existing self-improving methods.
The paper identifies a convergence issue in the Self-Improving Alignment (SAIL) algorithm due to non-strong concavity and proposes a regularized version (SAIL-RevKL) that satisfies the PL condition, achieving near-linear sample complexity and outperforming vanilla SAIL on MuJoCo and LLM tasks.
The Self-Improving Alignment (SAIL) algorithm addresses distribution shift by reducing a bilevel formulation of the problem to an efficient, single-level method. Empirically, SAIL has demonstrated strong performance on this task. However, a formal analysis of its convergence properties has been lacking. We identify a key theoretical challenge: the standard SAIL objective function is not guaranteed to be strongly concave due to unfavorable properties of its Hessian. To address this limitation, we propose a regularized objective, SAIL-RevKL, which incorporates a reverse Kullback-Leibler (KL) divergence penalty to improve the optimization landscape. Our central theoretical contribution is to prove that this regularized objective satisfies the Polyak-Lojasiewicz (PL) condition within a bounded parameter space. We establish global convergence guarantees, achieving a near-linear sample complexity. We further validate the effectiveness and stability of SAIL-RevKL through empirical evaluations, demonstrating that it outperforms the vanilla SAIL on both MuJoCo benchmarks and LLM alignment tasks.