RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models
This addresses a key optimization challenge for improving LLM performance on practical benchmarks, though it is an incremental advance in policy optimization methods.
The paper tackles the mismatch between risk-neutral training objectives and risk-seeking evaluation metrics like Pass@k and Max@k in large language models, proposing RSPO to directly optimize these metrics and addressing the 'hitchhiking' problem, with experimental results showing improved performance.
Current large language model post-training optimizes a risk-neutral objective that maximizes expected reward, yet evaluation relies heavily on risk-seeking metrics like Pass@k (at least one success in k trials) and Max@k (maximum reward across k responses). This mismatch in risk preferences can inevitably lead to suboptimal performance. To bridge this gap, we propose Risk-Seeking Policy Optimization (RSPO), a novel method that directly targets Pass@k and Max@k during training. A key challenge in optimizing these metrics is the "hitchhiking" problem: low-reward responses are inadvertently reinforced if they co-occur with a high-reward response within a sample of k generations, resulting in inefficient optimization. RSPO addresses this problem by leveraging the closed-form probability that a given response is the maximum among k samplings. Despite the complexity of nested gradients over multiple responses, RSPO produces efficient, unbiased gradient estimators for both metrics. We validate our approach with both rigorous theoretical analysis and comprehensive experimental results.