Power Law Trends in Speedrunning and Machine Learning
This provides incremental improvements in forecasting methods for speedrunning and ML benchmarks, primarily benefiting researchers in these domains.
The paper tackles the problem of forecasting speedrunning world record improvements by identifying power law patterns in their progression, achieving statistically significant improvements over baseline predictions with p < 10^-5. It extends this approach to machine learning benchmarks, showing they are far from saturation while suggesting sudden large improvements are unlikely.
We find that improvements in speedrunning world records follow a power law pattern. Using this observation, we answer an outstanding question from previous work: How do we improve on the baseline of predicting no improvement when forecasting speedrunning world records out to some time horizon, such as one month? Using a random effects model, we improve on this baseline for relative mean square error made on predicting out-of-sample world record improvements as the comparison metric at a $p < 10^{-5}$ significance level. The same set-up improves \textit{even} on the ex-post best exponential moving average forecasts at a $p = 0.15$ significance level while having access to substantially fewer data points. We demonstrate the effectiveness of this approach by applying it to Machine Learning benchmarks and achieving forecasts that exceed a baseline. Finally, we interpret the resulting model to suggest that 1) ML benchmarks are far from saturation and 2) sudden large improvements in Machine Learning are unlikely but cannot be ruled out.