AdaFish: Fast low-rank parameter-efficient fine-tuning by using
  second-order information
Jiang Hu Quanzheng Li
Recent advancements in large-scale pretrained models have significantly
improved performance across a variety of tasks in natural language processing
and computer vision. However, the extensive number of parameters in these
models necessitates substantial memory and computational resources for full
training. To adapt these models for downstream tasks or specific
application-oriented datasets, parameter-efficient fine-tuning methods
leveraging pretrained parameters have gained considerable attention. However,
it can still be time-consuming due to lots of parameters and epochs. In this
work, we introduce AdaFish, an efficient algorithm of the second-order type
designed to expedite the training process within low-rank decomposition-based
fine-tuning frameworks. Our key observation is that the associated generalized
Fisher information matrix is either low-rank or extremely small-scaled. Such a
generalized Fisher information matrix is shown to be equivalent to the Hessian
matrix. Moreover, we prove the global convergence of AdaFish, along with its
iteration/oracle complexity. Numerical experiments show that our algorithm is
quite competitive with the state-of-the-art AdamW method.