How predictable is language model benchmark performance?
This work addresses the challenge of forecasting AI capabilities for researchers and practitioners, though it is incremental in refining existing scaling laws.
The study tackled the problem of predicting language model benchmark performance by analyzing compute scaling across eleven architectures, finding that average benchmark performance is decently predictable with 6 percentage points error, while individual tasks are less predictable with 18 percentage points error.
We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.