The MultiBERTs: BERT Reproductions for Robustness Analysis
This work addresses the need for more reliable and generalizable findings in NLP research by providing tools to account for variability in model training, which is incremental but practical for researchers using pre-trained models.
The authors tackled the problem of drawing robust conclusions from single-checkpoint experiments in pre-trained models like BERT by introducing the MultiBERTs, a set of 25 BERT-Base checkpoints with varied initializations and data shuffling, and the Multi-Bootstrap method for statistical inference, enabling detection of effects such as gender bias in coreference resolution that might be missed otherwise.
Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternate strategy is needed to make principled statements about procedures. To enable researchers to draw more robust conclusions, we introduce the MultiBERTs, a set of 25 BERT-Base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data. We also define the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint. We release our models and statistical library along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.