A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks
This addresses the need for better multi-task learning methods in NLP, though it appears incremental as it builds on existing multi-task approaches with a hierarchical twist.
The paper tackles the problem of understanding when multi-task learning is effective for learning rich representations in NLP by introducing a hierarchical model trained on semantic tasks, achieving state-of-the-art results on Named Entity Recognition, Entity Mention Detection, and Relation Extraction without hand-engineered features or external tools.
Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.