On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval
This work addresses multimodal chatbots for research and industry, but it is incremental as it builds on existing methods for retrieval-based systems.
The paper tackles the problem of multimodal dialogue response retrieval by formulating it as three subtasks and proposing integration methods, finding that an end-to-end approach achieves comparable performance to a two-step method while a parameter sharing strategy reduces parameters and boosts performance.
Multimodal chatbots have become one of the major topics for dialogue systems in both research community and industry. Recently, researchers have shed light on the multimodality of responses as well as dialogue contexts. This work explores how a dialogue system can output responses in various modalities such as text and image. To this end, we first formulate a multimodal dialogue response retrieval task for retrieval-based systems as the combination of three subtasks. We then propose three integration methods based on a two-step approach and an end-to-end approach, and compare the merits and demerits of each method. Experimental results on two datasets demonstrate that the end-to-end approach achieves comparable performance without an intermediate step in the two-step approach. In addition, a parameter sharing strategy not only reduces the number of parameters but also boosts performance by transferring knowledge across the subtasks and the modalities.