PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model
Zheng Zhang Yeyao Ma Enming Zhang Xiang Bai
PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address
the segmentation task challenges. To overcome the limitation of the LMM being
limited to textual output, PSALM incorporates a mask decoder and a
well-designed input schema to handle a variety of segmentation tasks. This
schema includes images, task instructions, conditional prompts, and mask
tokens, which enable the model to generate and classify segmentation masks
effectively. The flexible design of PSALM supports joint training across
multiple datasets and tasks, leading to improved performance and task
generalization. PSALM achieves superior results on several benchmarks, such as
RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive,
and further exhibits zero-shot capabilities on unseen tasks, such as
open-vocabulary segmentation, generalized referring expression segmentation and
video object segmentation, making a significant step towards a GPT moment in
computer vision. Through extensive experiments, PSALM demonstrates its
potential to transform the domain of image segmentation, leveraging the robust
visual understanding capabilities of LMMs as seen in natural language
processing. Code and models are available at https://github.com/zamling/PSALM.