Multimodal Diffusion Segmentation Model for Object Segmentation from
  Manipulation Instructions
Yui Iioka Yu Yoshida Yuiga Wada Shumpei Hatanaka Komei Sugiura
In this study, we aim to develop a model that comprehends a natural language
instruction (e.g., "Go to the living room and get the nearest pillow to the
radio art on the wall") and generates a segmentation mask for the target
everyday object. The task is challenging because it requires (1) the
understanding of the referring expressions for multiple objects in the
instruction, (2) the prediction of the target phrase of the sentence among the
multiple phrases, and (3) the generation of pixel-wise segmentation masks
rather than bounding boxes. Studies have been conducted on languagebased
segmentation methods; however, they sometimes mask irrelevant regions for
complex sentences. In this paper, we propose the Multimodal Diffusion
Segmentation Model (MDSM), which generates a mask in the first stage and
refines it in the second stage. We introduce a crossmodal parallel feature
extraction mechanism and extend diffusion probabilistic models to handle
crossmodal features. To validate our model, we built a new dataset based on the
well-known Matterport3D and REVERIE datasets. This dataset consists of
instructions with complex referring expressions accompanied by real indoor
environmental images that feature various target objects, in addition to
pixel-wise segmentation masks. The performance of MDSM surpassed that of the
baseline method by a large margin of +10.13 mean IoU.