BAdam: A Memory Efficient Full Parameter Training Method for Large
  Language Models
Qijun Luo Hengxu Yu Xiao Li
This work presents BAdam, an optimizer that leverages the block coordinate
optimization framework with Adam as the inner solver. BAdam offers a memory
efficient approach to the full parameter finetuning of large language models
and reduces running time of the backward process thanks to the chain rule
property. Experimentally, we apply BAdam to instruction-tune the Llama 2-7B
model on the Alpaca-GPT4 dataset using a single RTX3090-24GB GPU. The results
indicate that BAdam exhibits superior convergence behavior in comparison to
LoRA and LOMO. Furthermore, our downstream performance evaluation of the
instruction-tuned models using the MT-bench shows that BAdam modestly surpasses
LoRA and more substantially outperforms LOMO. Finally, we compare BAdam with
Adam on a medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE
benchmark. The results demonstrate that BAdam is capable of narrowing the
performance gap with Adam. Our code is available at
https://github.com/Ledzy/BAdam.