Measuring the Instability of Fine-Tuning
This work addresses the need for better instability measurements in machine learning, particularly for researchers and practitioners dealing with fine-tuning on small datasets, but it is incremental as it builds on prior studies without introducing new mitigation techniques.
The paper tackled the problem of measuring instability in fine-tuning pre-trained language models, analyzing seven measures beyond standard deviation and proposing a framework to evaluate their validity, with results showing differences in how these measures assess existing mitigation methods.
Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets. Many previous studies have investigated this instability and proposed methods to mitigate it. However, most studies only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. In this paper, we analyze SD and six other measures quantifying instability at different levels of granularity. Moreover, we propose a systematic framework to evaluate the validity of these measures. Finally, we analyze the consistency and difference between different measures by reassessing existing instability mitigation methods. We hope our results will inform the development of better measurements of fine-tuning instability.