LaSagnA: Language-based Segmentation Assistant for Complex Queries
Cong Wei Haoxian Tan Yujie Zhong Yujiu Yang Lin Ma
Recent advancements have empowered Large Language Models for Vision (vLLMs)
to generate detailed perceptual outcomes, including bounding boxes and masks.
Nonetheless, there are two constraints that restrict the further application of
these vLLMs: the incapability of handling multiple targets per query and the
failure to identify the absence of query objects in the image. In this study,
we acknowledge that the main cause of these problems is the insufficient
complexity of training queries. Consequently, we define the general sequence
format for complex queries. Then we incorporate a semantic segmentation task in
the current pipeline to fulfill the requirements of training data. Furthermore,
we present three novel strategies to effectively handle the challenges arising
from the direct integration of the proposed format. The effectiveness of our
model in processing complex queries is validated by the comparable results with
conventional methods on both close-set and open-set semantic segmentation
datasets. Additionally, we outperform a series of vLLMs in reasoning and
referring segmentation, showcasing our model's remarkable capabilities. We
release the code at https://github.com/congvvc/LaSagnA.