Selective Feature Adapter for Dense Vision Transformers
Xueqing Deng Qi Fan Xiaojie Jin Linjie Yang Peng Wang
Fine-tuning pre-trained transformer models, e.g., Swin Transformer, are
successful in numerous downstream for dense prediction vision tasks. However,
one major issue is the cost/storage of their huge amount of parameters, which
becomes increasingly challenging to handle with the growing amount of vision
tasks. In this paper, we propose an effective approach to alleviate the issue,
namely selective feature adapter (SFA). It achieves state-of-the-art (SoTA)
performance under any given budget of trainable parameters, and demonstrates
comparable or better performance than fully fine-tuned models across various
dense tasks. Specifically, SFA consists of external adapters and internal
adapters which are sequentially operated over a transformer model. For external
adapters, we properly select the places and amount of additional multilayer
perception (MLP). For internal adapters, we transform a few task-important
parameters inside the transformer, which are automatically discovered through a
simple yet effective lottery ticket algorithm. Our experiments show that the
dual adapter module, a.k.a SFA, is essential to achieve the best trade-off on
dense vision tasks, such as segmentation, detection and depth-estimation,
outperforming other adapters with a single module.