Prototypical Contrastive Transfer Learning for Multimodal Language
  Understanding
Seitaro Otsuki Shintaro Ishikawa Komei Sugiura
Although domestic service robots are expected to assist individuals who
require support, they cannot currently interact smoothly with people through
natural language. For example, given the instruction "Bring me a bottle from
the kitchen," it is difficult for such robots to specify the bottle in an
indoor environment. Most conventional models have been trained on real-world
datasets that are labor-intensive to collect, and they have not fully leveraged
simulation data through a transfer learning framework. In this study, we
propose a novel transfer learning approach for multimodal language
understanding called Prototypical Contrastive Transfer Learning (PCTL), which
uses a new contrastive loss called Dual ProtoNCE. We introduce PCTL to the task
of identifying target objects in domestic environments according to free-form
natural language instructions. To validate PCTL, we built new real-world and
simulation datasets. Our experiment demonstrated that PCTL outperformed
existing methods. Specifically, PCTL achieved an accuracy of 78.1%, whereas
simple fine-tuning achieved an accuracy of 73.4%.