BURT: BERT-inspired Universal Representation from Twin Structure
This addresses the limitation of existing pre-trained models like BERT in handling diverse text granularities, offering a more versatile representation method for NLP tasks, though it is incremental as it builds on BERT and Siamese networks.
The paper tackles the problem of generating universal fixed-size representations for text sequences of any granularity (words, phrases, sentences) by proposing BURT, which uses a Siamese network with multiple training objectives on natural language inference and paraphrase data, resulting in substantial outperformance on sentence-level datasets and significant improvements in word/phrase-level representation tasks.
Pre-trained contextualized language models such as BERT have shown great effectiveness in a wide range of downstream Natural Language Processing (NLP) tasks. However, the effective representations offered by the models target at each token inside a sequence rather than each sequence and the fine-tuning step involves the input of both sequences at one time, leading to unsatisfying representations of various sequences with different granularities. Especially, as sentence-level representations taken as the full training context in these models, there comes inferior performance on lower-level linguistic units (phrases and words). In this work, we present BURT (BERT inspired Universal Representation from Twin Structure) that is capable of generating universal, fixed-size representations for input sequences of any granularity, i.e., words, phrases, and sentences, using a large scale of natural language inference and paraphrase data with multiple training objectives. Our proposed BURT adopts the Siamese network, learning sentence-level representations from natural language inference dataset and word/phrase-level representations from paraphrasing dataset, respectively. We evaluate BURT across different granularities of text similarity tasks, including STS tasks, SemEval2013 Task 5(a) and some commonly used word similarity tasks, where BURT substantially outperforms other representation models on sentence-level datasets and achieves significant improvements in word/phrase-level representation.