BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
This work addresses the interpretability and performance enhancement of BERT for NLP practitioners, though it is incremental as it builds on existing BERT architectures.
The authors discovered that BERT's output layer can reconstruct input sentences from any hidden layer, a phenomenon that holds across various BERT-based models, and leveraged this by duplicating layers to create deeper models, which improved performance in downstream tasks after fine-tuning.
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box. A variety of previous works have tried to lift the veil of BERT and understand each layer's functionality. In this paper, we found that surprisingly the output layer of BERT can reconstruct the input sentence by directly taking each layer of BERT as input, even though the output layer has never seen the input other than the final hidden layer. This fact remains true across a wide variety of BERT-based models, even when some layers are duplicated. Based on this observation, we propose a quite simple method to boost the performance of BERT. By duplicating some layers in the BERT-based models to make it deeper (no extra training required in this step), they obtain better performance in the downstream tasks after fine-tuning.