Injecting Hierarchy with U-Net Transformers
This work addresses a limitation in Transformer models for NLP, specifically in dialogue tasks, but it appears incremental as it adapts an existing computer vision method to a new domain.
The authors tackled the problem of Transformers lacking explicit hierarchical structure in language by introducing hierarchical processing inspired by U-Net, and they demonstrated that this architecture outperforms vanilla Transformers and strong baselines in chit-chat dialogue.
The Transformer architecture has become increasingly popular over the past two years, owing to its impressive performance on a number of natural language processing (NLP) tasks. However, all Transformer computations occur at the level of word representations and therefore, it may be argued that Transformer models do not explicitly attempt to learn hierarchical structure which is widely assumed to be integral to language. In the present work, we introduce hierarchical processing into the Transformer model, taking inspiration from the U-Net architecture, popular in computer vision for its hierarchical view of natural images. We empirically demonstrate that the proposed architecture outperforms both the vanilla Transformer and some strong baselines in the domain of chit-chat dialogue.