One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware
  Quantization Training
Lianbo Ma Yuee Zhou Jianlun Ma Guo Yu Qing Li
Weight quantization is an effective technique to compress deep neural
networks for their deployment on edge devices with limited resources.
Traditional loss-aware quantization methods commonly use the quantized gradient
to replace the full-precision gradient. However, we discover that the gradient
error will lead to an unexpected zig-zagging-like issue in the gradient descent
learning procedures, where the gradient directions rapidly oscillate or
zig-zag, and such issue seriously slows down the model convergence.
Accordingly, this paper proposes a one-step forward and backtrack way for
loss-aware quantization to get more accurate and stable gradient direction to
defy this issue. During the gradient descent learning, a one-step forward
search is designed to find the trial gradient of the next-step, which is
adopted to adjust the gradient of current step towards the direction of fast
convergence. After that, we backtrack the current step to update the
full-precision and quantized weights through the current-step gradient and the
trial gradient. A series of theoretical analysis and experiments on benchmark
deep models have demonstrated the effectiveness and competitiveness of the
proposed method, and our method especially outperforms others on the
convergence performance.