Application of frozen large-scale models to multimodal task-oriented
  dialogue
Tatsuki Kawamoto Takuma Suzuki Ko Miyama Takumi Meguro Tomohiro Takagi
In this study, we use the existing Large Language Models ENnhanced to See
Framework (LENS Framework) to test the feasibility of multimodal task-oriented
dialogues. The LENS Framework has been proposed as a method to solve computer
vision tasks without additional training and with fixed parameters of
pre-trained models. We used the Multimodal Dialogs (MMD) dataset, a multimodal
task-oriented dialogue benchmark dataset from the fashion field, and for the
evaluation, we used the ChatGPT-based G-EVAL, which only accepts textual
modalities, with arrangements to handle multimodal data. Compared to
Transformer-based models in previous studies, our method demonstrated an
absolute lift of 10.8% in fluency, 8.8% in usefulness, and 5.2% in relevance
and coherence. The results show that using large-scale models with fixed
parameters rather than using models trained on a dataset from scratch improves
performance in multimodal task-oriented dialogues. At the same time, we show
that Large Language Models (LLMs) are effective for multimodal task-oriented
dialogues. This is expected to lead to efficient applications to existing
systems.