Catalog Phrase Grounding (CPG): Grounding of Product Textual Attributes
  in Product Images for e-commerce Vision-Language Applications
Wenyi Wu Karim Bouyarmane Ismail Tutar
We present Catalog Phrase Grounding (CPG), a model that can associate product
textual data (title, brands) into corresponding regions of product images
(isolated product region, brand logo region) for e-commerce vision-language
applications. We use a state-of-the-art modulated multimodal transformer
encoder-decoder architecture unifying object detection and phrase-grounding. We
train the model in self-supervised fashion with 2.3 million image-text pairs
synthesized from an e-commerce site. The self-supervision data is annotated
with high-confidence pseudo-labels generated with a combination of teacher
models: a pre-trained general domain phrase grounding model (e.g. MDETR) and a
specialized logo detection model. This allows CPG, as a student model, to
benefit from transfer knowledge from these base models combining general-domain
knowledge and specialized knowledge. Beyond immediate catalog phrase grounding
tasks, we can benefit from CPG representations by incorporating them as ML
features into downstream catalog applications that require deep semantic
understanding of products. Our experiments on product-brand matching, a
challenging e-commerce application, show that incorporating CPG representations
into the existing production ensemble system leads to on average 5% recall
improvement across all countries globally (with the largest lift of 11% in a
single country) at fixed 95% precision, outperforming other alternatives
including a logo detection teacher model and ResNet50.