Quantized Approximately Orthogonal Recurrent Neural Networks
Armand Foucault Franck Mamalet François Malgouyres
Orthogonal recurrent neural networks (ORNNs) are an appealing option for
learning tasks involving time series with long-term dependencies, thanks to
their simplicity and computational stability. However, these networks often
require a substantial number of parameters to perform well, which can be
prohibitive in power-constrained environments, such as compact devices. One
approach to address this issue is neural network quantization. The construction
of such networks remains an open problem, acknowledged for its inherent
instability.In this paper, we explore the quantization of the recurrent and
input weight matrices in ORNNs, leading to Quantized approximately Orthogonal
RNNs (QORNNs). We investigate one post-training quantization (PTQ) strategy and
three quantization-aware training (QAT) algorithms that incorporate orthogonal
constraints and quantized weights. Empirical results demonstrate the advantages
of employing QAT over PTQ. The most efficient model achieves results similar to
state-of-the-art full-precision ORNN and LSTM on a variety of standard
benchmarks, even with 3-bits quantization.