HUBERT Untangles BERT to Improve Transfer across NLP Tasks
This addresses transfer learning challenges in NLP, but appears incremental as it builds on existing BERT and TPR methods.
The paper tackles the problem of improving transfer learning across NLP tasks by introducing HUBERT, which combines Tensor-Product Representations with BERT to untangle data-specific semantics from general language structure. The result shows effectiveness validated on the GLUE benchmark and HANS dataset, though no concrete numbers are provided.
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.