Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
This addresses a key limitation in applying scaling laws to predict downstream task performance for language model researchers, though it is an incremental improvement on existing methods.
The paper tackles the gap between validation loss and downstream task performance in scaling laws for language models by introducing the Capability Salience Vector, which assigns importance weights to tokens to align loss with capabilities, resulting in significantly improved predictability on various benchmarks.
Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model's downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model's capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks.