LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation
Hongyun Zhou Xiangyu Lu Wang Xu Conghui Zhu Tiejun Zhao
Low-Rank Adaptation (LoRA) introduces auxiliary parameters for each layer to
fine-tune the pre-trained model under limited computing resources. But it still
faces challenges of resource consumption when scaling up to larger models.
Previous studies employ pruning techniques by evaluating the importance of LoRA
parameters for different layers to address the problem. However, these efforts
only analyzed parameter features to evaluate their importance. Indeed, the
output of LoRA related to the parameters and data is the factor that directly
impacts the frozen model. To this end, we propose LoRA-drop which evaluates the
importance of the parameters by analyzing the LoRA output. We retain LoRA for
important layers and the LoRA of the other layers share the same parameters.
Abundant experiments on NLU and NLG tasks demonstrate the effectiveness of
LoRA-drop.