Synthesizing Knowledge-enhanced Features for Real-world Zero-shot Food
  Detection
Pengfei Zhou Weiqing Min Jiajun Song Yang Zhang Shuqiang Jiang
Food computing brings various perspectives to computer vision like
vision-based food analysis for nutrition and health. As a fundamental task in
food computing, food detection needs Zero-Shot Detection (ZSD) on novel unseen
food objects to support real-world scenarios, such as intelligent kitchens and
smart restaurants. Therefore, we first benchmark the task of Zero-Shot Food
Detection (ZSFD) by introducing FOWA dataset with rich attribute annotations.
Unlike ZSD, fine-grained problems in ZSFD like inter-class similarity make
synthesized features inseparable. The complexity of food semantic attributes
further makes it more difficult for current ZSD methods to distinguish various
food categories. To address these problems, we propose a novel framework ZSFDet
to tackle fine-grained problems by exploiting the interaction between complex
attributes. Specifically, we model the correlation between food categories and
attributes in ZSFDet by multi-source graphs to provide prior knowledge for
distinguishing fine-grained features. Within ZSFDet, Knowledge-Enhanced Feature
Synthesizer (KEFS) learns knowledge representation from multiple sources (e.g.,
ingredients correlation from knowledge graph) via the multi-source graph
fusion. Conditioned on the fusion of semantic knowledge representation, the
region feature diffusion model in KEFS can generate fine-grained features for
training the effective zero-shot detector. Extensive evaluations demonstrate
the superior performance of our method ZSFDet on FOWA and the widely-used food
dataset UECFOOD-256, with significant improvements by 1.8% and 3.7% ZSD mAP
compared with the strong baseline RRFS. Further experiments on PASCAL VOC and
MS COCO prove that enhancement of the semantic knowledge can also improve the
performance on general ZSD. Code and dataset are available at
https://github.com/LanceZPF/KEFS.