Rethinking GSPO: The Perplexity-Entropy Equivalence
This provides a theoretical insight for researchers working on policy gradient methods in reinforcement learning, but it is incremental as it reinterprets an existing algorithm without introducing new methods.
The paper reinterprets GSPO's importance ratios by linking them to perplexity and cross-entropy, showing that sequence-level weights equal inverse perplexity ratios and exponential cross-entropy changes, which helps explain GSPO's empirical properties like variance reduction and stability in training mixture-of-experts models.
We provide a new perspective on GSPO's length-normalized importance ratios by establishing their connection to information-theoretic quantities. We show that GSPO's sequence-level weight $s(θ) = (π_θ/π_{θ_{\text{old}}})^{1/|y|}$ can be equivalently expressed as the inverse perplexity ratio $\text{PPL}_{θ_{\text{old}}}/\text{PPL}_θ$ and as the exponential cross-entropy change $\exp(ΔH)$. While the perplexity-entropy relationship follows from standard definitions, this observation provides a useful lens for understanding GSPO: the algorithm weights policy gradient updates by perplexity ratios, offering an information-theoretic interpretation of the importance weights. This perspective helps explain GSPO's empirical properties, including log-domain variance reduction through geometric averaging and stability in training mixture-of-experts models. We validate the mathematical equivalences and variance predictions through controlled experiments on mathematical reasoning tasks.