FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision
  Transformers
Zheng Yuan Jie Zhang Shiguang Shan
In recent years, the Vision Transformer (ViT) model has gradually become
mainstream in various computer vision tasks, and the robustness of the model
has received increasing attention. However, existing large models tend to
prioritize performance during training, potentially neglecting the robustness,
which may lead to serious security concerns. In this paper, we establish a new
challenge: exploring how to use a small number of additional parameters for
adversarial finetuning to quickly and effectively enhance the adversarial
robustness of a standardly trained model. To address this challenge, we develop
the novel LNLoRA module, incorporating a learnable layer normalization before
the conventional LoRA module, which helps mitigate magnitude differences in
parameters between the adversarial and standard training paradigms.
  Furthermore, we propose the FullLoRA-AT framework by integrating the
learnable LNLoRA modules into all key components of ViT-based models while
keeping the pretrained model frozen, which can significantly improve the model
robustness via adversarial finetuning in a parameter-efficient manner.
  Extensive experiments on CIFAR-10, CIFAR-100, and Imagenette demonstrate the
superiority of our proposed FullLoRA-AT framework. It achieves comparable
robustness with full finetuning while only requiring about 5% of the learnable
parameters. This also effectively addresses concerns regarding extra model
storage space and enormous training time caused by adversarial finetuning.