MaskConver: Revisiting Pure Convolution Model for Panoptic Segmentation
Abdullah Rashwan Jiageng Zhang Ali Taalimi Fan Yang Xingyi Zhou Chaochao Yan Liang-Chieh Chen Yeqing Li
In recent years, transformer-based models have dominated panoptic
segmentation, thanks to their strong modeling capabilities and their unified
representation for both semantic and instance classes as global binary masks.
In this paper, we revisit pure convolution model and propose a novel panoptic
architecture named MaskConver. MaskConver proposes to fully unify things and
stuff representation by predicting their centers. To that extent, it creates a
lightweight class embedding module that can break the ties when multiple
centers co-exist in the same location. Furthermore, our study shows that the
decoder design is critical in ensuring that the model has sufficient context
for accurate detection and segmentation. We introduce a powerful ConvNeXt-UNet
decoder that closes the performance gap between convolution- and
transformerbased models. With ResNet50 backbone, our MaskConver achieves 53.6%
PQ on the COCO panoptic val set, outperforming the modern convolution-based
model, Panoptic FCN, by 9.3% as well as transformer-based models such as
Mask2Former (+1.7% PQ) and kMaX-DeepLab (+0.6% PQ). Additionally, MaskConver
with a MobileNet backbone reaches 37.2% PQ, improving over Panoptic-DeepLab by
+6.4% under the same FLOPs/latency constraints. A further optimized version of
MaskConver achieves 29.7% PQ, while running in real-time on mobile devices. The
code and model weights will be publicly available