ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with
  Visual Prompt Tuning
Beomyoung Kim Joonsang Yu Sung Ju Hwang
Panoptic segmentation, combining semantic and instance segmentation, stands
as a cutting-edge computer vision task. Despite recent progress with deep
learning models, the dynamic nature of real-world applications necessitates
continual learning, where models adapt to new classes (plasticity) over time
without forgetting old ones (catastrophic forgetting). Current continual
segmentation methods often rely on distillation strategies like knowledge
distillation and pseudo-labeling, which are effective but result in increased
training complexity and computational overhead. In this paper, we introduce a
novel and efficient method for continual panoptic segmentation based on Visual
Prompt Tuning, dubbed ECLIPSE. Our approach involves freezing the base model
parameters and fine-tuning only a small set of prompt embeddings, addressing
both catastrophic forgetting and plasticity and significantly reducing the
trainable parameters. To mitigate inherent challenges such as error propagation
and semantic drift in continual segmentation, we propose logit manipulation to
effectively leverage common knowledge across the classes. Experiments on ADE20K
continual panoptic segmentation benchmark demonstrate the superiority of
ECLIPSE, notably its robustness against catastrophic forgetting and its
reasonable plasticity, achieving a new state-of-the-art. The code is available
at https://github.com/clovaai/ECLIPSE.