Visualizing and Understanding the Effectiveness of BERT
This addresses the problem of understanding BERT's effectiveness for NLP researchers, but it is incremental as it visualizes known phenomena without introducing new methods.
The paper investigates why BERT's pre-training and fine-tuning improves performance and generalization in NLP tasks, finding that pre-training provides good initial points leading to wider optima and robust fine-tuning, with lower layers learning more transferable representations.
Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks. However, it is unclear why the pre-training-then-fine-tuning paradigm can improve performance and generalization capability across different tasks. In this paper, we propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. First, we find that pre-training reaches a good initial point across downstream tasks, which leads to wider optima and easier optimization compared with training from scratch. We also demonstrate that the fine-tuning procedure is robust to overfitting, even though BERT is highly over-parameterized for downstream tasks. Second, the visualization results indicate that fine-tuning BERT tends to generalize better because of the flat and wide optima, and the consistency between the training loss surface and the generalization error surface. Third, the lower layers of BERT are more invariant during fine-tuning, which suggests that the layers that are close to input learn more transferable representations of language.