A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction
For practitioners needing to predict LLM fine-tuning performance without actual fine-tuning, this work provides a theoretical framework and practical probing strategy to reduce costs.
This paper establishes theoretical limits on pre-hoc fine-tuning prediction, decomposing prediction risk into an intrinsic limit and a reducible optimization variance with a necessary lower bound on its decay rate. It introduces a budget-optimal probing principle and a predictability phase diagram, validated on synthetic and real-world benchmarks.
The high cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense. However, the theoretical limits of pre-hoc performance prediction remain unexplored. We formulate it as a stochastic estimation problem under information constraints, decomposing prediction risk into two components: an intrinsic limit (static data-model compatibility) and a reducible optimization variance. We prove that optimization variance admits a necessary lower bound on its decay rate, implying fundamental constraints on how quickly uncertainty dissipates, regardless of the predictor used. Based on these dynamics, we derive a budget-optimal probing principle and introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. Extensive experiments on synthetic and real-world benchmarks validate these theoretical regimes and demonstrate the efficiency of our probing strategy.