Stochastic Two Points Method for Deep Model Zeroth-order Optimization
Yijiang Pang Jiayu Zhou
Large foundation models, such as large language models, have performed
exceptionally well in various application scenarios. Building or fully
fine-tuning such large models is usually prohibitive due to either hardware
budget or lack of access to backpropagation. The zeroth-order methods offer a
promising direction for tackling this challenge, where only forward passes are
needed to update the model. This paper introduces an efficient Stochastic
Two-Point (S2P) approach within the gradient-free regime. We present the
theoretical convergence properties of S2P under the general and relaxed
smoothness assumptions. The theoretical properties also shed light on a faster
and more stable S2P variant, Accelerated S2P (AS2P), through exploiting our new
convergence properties that better represent the dynamics of deep models in
training. Our comprehensive empirical results show that AS2P is highly
effective in optimizing objectives for large deep models, including language
models, and outperforms standard methods across various model types and scales,
with 2 $\times$ speed-up in training over most conducted tasks.