Learning without Forgetting for Vision-Language Models
Da-Wei Zhou Yuanhan Zhang Jingyi Ning Han-Jia Ye De-Chuan Zhan Ziwei Liu
Class-Incremental Learning (CIL) or continual learning is a desired
capability in the real world, which requires a learning system to adapt to new
tasks without forgetting former ones. While traditional CIL methods focus on
visual information to grasp core features, recent advances in Vision-Language
Models (VLM) have shown promising capabilities in learning generalizable
representations with the aid of textual information. However, when continually
trained with new classes, VLMs often suffer from catastrophic forgetting of
former knowledge. Applying VLMs to CIL poses two major challenges: 1) how to
adapt the model without forgetting; and 2) how to make full use of the
multi-modal information. To this end, we propose PROjectiOn Fusion (PROOF) that
enables VLMs to learn without forgetting. To handle the first challenge, we
propose training task-specific projections based on the frozen image/text
encoders. When facing new tasks, new projections are expanded and former
projections are fixed, alleviating the forgetting of old concepts. For the
second challenge, we propose the fusion module to better utilize the
cross-modality information. By jointly adjusting visual and textual features,
the model can capture semantic information with stronger representation
ability. Extensive experiments on nine benchmark datasets validate PROOF
achieves state-of-the-art performance.