Latent Universal Task-Specific BERT
This work addresses the need for adaptable language models in specific domains like social media, though it is incremental as it builds on existing BERT and Universal Transformer frameworks.
The paper tackles the problem of creating a language representation model that is both general and domain-specific by combining BERT and Universal Transformer mechanisms, adding a latent variable for writer persona and topics, and introducing a method to trade off generalization for domain-specific utility, resulting in a pre-trained model on 100 million tweets.
This paper describes a language representation model which combines the Bidirectional Encoder Representations from Transformers (BERT) learning mechanism described in Devlin et al. (2018) with a generalization of the Universal Transformer model described in Dehghani et al. (2018). We further improve this model by adding a latent variable that represents the persona and topics of interests of the writer for each training example. We also describe a simple method to improve the usefulness of our language representation for solving problems in a specific domain at the expense of its ability to generalize to other fields. Finally, we release a pre-trained language representation model for social texts that was trained on 100 million tweets.