FoodSAM: Any Food Segmentation
Xing Lan Jiayi Lyu Hanyu Jiang Kun Dong Zehai Niu Yi Zhang Jian Xue
In this paper, we explore the zero-shot capability of the Segment Anything
Model (SAM) for food image segmentation. To address the lack of class-specific
information in SAM-generated masks, we propose a novel framework, called
FoodSAM. This innovative approach integrates the coarse semantic mask with
SAM-generated masks to enhance semantic segmentation quality. Besides, we
recognize that the ingredients in food can be supposed as independent
individuals, which motivated us to perform instance segmentation on food
images. Furthermore, FoodSAM extends its zero-shot capability to encompass
panoptic segmentation by incorporating an object detector, which renders
FoodSAM to effectively capture non-food object information. Drawing inspiration
from the recent success of promptable segmentation, we also extend FoodSAM to
promptable segmentation, supporting various prompt variants. Consequently,
FoodSAM emerges as an all-encompassing solution capable of segmenting food
items at multiple levels of granularity. Remarkably, this pioneering framework
stands as the first-ever work to achieve instance, panoptic, and promptable
segmentation on food images. Extensive experiments demonstrate the feasibility
and impressing performance of FoodSAM, validating SAM's potential as a
prominent and influential tool within the domain of food image segmentation. We
release our code at https://github.com/jamesjg/FoodSAM.