Application of frozen large-scale models to multimodal task-oriented dialogue
This work addresses efficient application of large-scale models to multimodal task-oriented dialogues in fashion, showing incremental improvements over existing methods.
The study applied the frozen LENS Framework to multimodal task-oriented dialogues using the MMD dataset, achieving absolute lifts of 10.8% in fluency, 8.8% in usefulness, and 5.2% in relevance and coherence compared to Transformer-based models.
In this study, we use the existing Large Language Models ENnhanced to See Framework (LENS Framework) to test the feasibility of multimodal task-oriented dialogues. The LENS Framework has been proposed as a method to solve computer vision tasks without additional training and with fixed parameters of pre-trained models. We used the Multimodal Dialogs (MMD) dataset, a multimodal task-oriented dialogue benchmark dataset from the fashion field, and for the evaluation, we used the ChatGPT-based G-EVAL, which only accepts textual modalities, with arrangements to handle multimodal data. Compared to Transformer-based models in previous studies, our method demonstrated an absolute lift of 10.8% in fluency, 8.8% in usefulness, and 5.2% in relevance and coherence. The results show that using large-scale models with fixed parameters rather than using models trained on a dataset from scratch improves performance in multimodal task-oriented dialogues. At the same time, we show that Large Language Models (LLMs) are effective for multimodal task-oriented dialogues. This is expected to lead to efficient applications to existing systems.