Learning Multi-Level Information for Dialogue Response Selection by Highway Recurrent Transformer
This work addresses the dialogue response selection task for conversational AI systems, presenting an incremental improvement with a novel attention mechanism.
The paper tackled the problem of selecting the most probable next sentence in dialogue contexts by proposing a Highway Recurrent Transformer model, which achieved capability in modeling both utterance-level and dialogue-level information as demonstrated in experiments on the DSTC7 dataset.
With the increasing research interest in dialogue response generation, there is an emerging branch formulating this task as selecting next sentences, where given the partial dialogue contexts, the goal is to determine the most probable next sentence. Following the recent success of the Transformer model, this paper proposes (1) a new variant of attention mechanism based on multi-head attention, called highway attention, and (2) a recurrent model based on transformer and the proposed highway attention, so-called Highway Recurrent Transformer. Experiments on the response selection task in the seventh Dialog System Technology Challenge (DSTC7) show the capability of the proposed model of modeling both utterance-level and dialogue-level information; the effectiveness of each module is further analyzed as well.