Detecting Concrete Visual Tokens for Multimodal Machine Translation
Braeden Bowen Vipin Vijayan Scott Grigsby Timothy Anderson Jeremy Gwinnup
The challenge of visual grounding and masking in multimodal machine
translation (MMT) systems has encouraged varying approaches to the detection
and selection of visually-grounded text tokens for masking. We introduce new
methods for detection of visually and contextually relevant (concrete) tokens
from source sentences, including detection with natural language processing
(NLP), detection with object detection, and a joint detection-verification
technique. We also introduce new methods for selection of detected tokens,
including shortest $n$ tokens, longest $n$ tokens, and all detected concrete
tokens. We utilize the GRAM MMT architecture to train models against
synthetically collated multimodal datasets of source images with masked
sentences, showing performance improvements and improved usage of visual
context during translation tasks over the baseline model.