OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
For practitioners in reinforcement learning and optimization who need to control distributional properties like risk or robustness, OrderGrad provides a unified, plug-and-play method to optimize order-statistic objectives.
OrderGrad introduces a family of unbiased gradient estimators for order-statistic objectives (e.g., VaR, CVaR, trimmed means, top-k) that optimize beyond the expected return. It is evaluated on LLM math post-training and other tasks, showing effectiveness where mean optimization is mismatched to deployment objectives.
Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-of-K discovery. We introduce OrderGrad, a family of likelihood-ratio and reparameterization gradient estimators for order-statistic objectives. OrderGrad optimizes finite-sample L-statistics, i.e., weighted averages of sorted rewards or costs, recovering objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights. For any fixed sample size and rank-weight vector, OrderGrad provides an unbiased gradient estimator for the corresponding order-statistic objective. The method is implemented as a simple reward transformation that can then be used in an otherwise standard policy-gradient or reparameterized update. We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective, including LLM math post-training and other tasks. OrderGrad provides a unified, plug-and-play route to risk-averse, robust, and exploratory learning. Code: https://github.com/paavo5/ordergrad