DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping
  Backward Propagation
Sunghyeon Woo Baeseong Park Byeongwook Kim Minjung Jo Sejung Kwon Dongsuk Jeon Dongsoo Lee
Training deep neural networks typically involves substantial computational
costs during both forward and backward propagation. The conventional layer
dropping techniques drop certain layers during training for reducing the
computations burden. However, dropping layers during forward propagation
adversely affects the training process by degrading accuracy. In this paper, we
propose Dropping Backward Propagation (DropBP), a novel approach designed to
reduce computational costs while maintaining accuracy. DropBP randomly drops
layers during the backward propagation, which does not deviate forward
propagation. Moreover, DropBP calculates the sensitivity of each layer to
assign appropriate drop rate, thereby stabilizing the training process. DropBP
is designed to enhance the efficiency of the training process with
backpropagation, thereby enabling the acceleration of both full fine-tuning and
parameter-efficient fine-tuning using backpropagation. Specifically, utilizing
DropBP in QLoRA reduces training time by 44%, increases the convergence speed
to the identical loss level by 1.5$\times$, and enables training with a
6.2$\times$ larger sequence length on a single NVIDIA-A100 80GiB GPU in
LLaMA2-70B. The code is available at https://github.com/WooSunghyeon/dropbp.