Off-Policy Learning in Large Action Spaces: Optimization Matters More Than Estimation
This addresses a practical obstacle in offline contextual bandits for decision-making applications, highlighting an incremental shift in focus from estimator optimization to optimization landscapes.
The paper tackles the problem of off-policy learning in large action spaces by showing that current methods face severe optimization issues, and demonstrates that simpler weighted log-likelihood objectives recover competitive or superior policies with better optimization properties.
Off-policy evaluation (OPE) and off-policy learning (OPL) are foundational for decision-making in offline contextual bandits. Recent advances in OPL primarily optimize OPE estimators with improved statistical properties, assuming that better estimators inherently yield superior policies. Although theoretically justified, we argue this estimator-centric approach neglects a critical practical obstacle: challenging optimization landscapes. In this paper, we provide theoretical insights and extensive empirical evidence showing that current OPL methods encounter severe optimization issues, particularly as action spaces become large. We demonstrate that simpler weighted log-likelihood objectives enjoy substantially better optimization properties and still recover competitive, often superior, learned policies. Our findings emphasize the necessity of explicitly addressing optimization considerations in the development of OPL algorithms for large action spaces.