Efficient Competitions and Online Learning with Strategic Forecasters
This work addresses incentive issues in forecasting competitions for machine learning practitioners, offering a more efficient solution.
The paper tackles the problem of distorted incentives in winner-take-all forecasting competitions by showing that the ELF mechanism requires Θ(n log n) events to select a near-optimal forecaster, while standard online learning algorithms achieve this with only O(log(n)/ε²) events, matching the optimal nonstrategic bound.
Winner-take-all competitions in forecasting and machine-learning suffer from distorted incentives. Witkowski et al. 2018 identified this problem and proposed ELF, a truthful mechanism to select a winner. We show that, from a pool of $n$ forecasters, ELF requires $Θ(n\log n)$ events or test data points to select a near-optimal forecaster with high probability. We then show that standard online learning algorithms select an $ε$-optimal forecaster using only $O(\log(n) / ε^2)$ events, by way of a strong approximate-truthfulness guarantee. This bound matches the best possible even in the nonstrategic setting. We then apply these mechanisms to obtain the first no-regret guarantee for non-myopic strategic experts.