ReLU and Addition-based Gated RNN
Rickard Brännvall Henrik Forsgren Fredrik Sandin Marcus Liwicki
We replace the multiplication and sigmoid function of the conventional
recurrent gate with addition and ReLU activation. This mechanism is designed to
maintain long-term memory for sequence processing but at a reduced
computational cost, thereby opening up for more efficient execution or larger
models on restricted hardware. Recurrent Neural Networks (RNNs) with gating
mechanisms such as LSTM and GRU have been widely successful in learning from
sequential data due to their ability to capture long-term dependencies.
Conventionally, the update based on current inputs and the previous state
history is each multiplied with dynamic weights and combined to compute the
next state. However, multiplication can be computationally expensive,
especially for certain hardware architectures or alternative arithmetic systems
such as homomorphic encryption. It is demonstrated that the novel gating
mechanism can capture long-term dependencies for a standard synthetic sequence
learning task while significantly reducing computational costs such that
execution time is reduced by half on CPU and by one-third under encryption.
Experimental results on handwritten text recognition tasks furthermore show
that the proposed architecture can be trained to achieve comparable accuracy to
conventional GRU and LSTM baselines. The gating mechanism introduced in this
paper may enable privacy-preserving AI applications operating under homomorphic
encryption by avoiding the multiplication of encrypted variables. It can also
support quantization in (unencrypted) plaintext applications, with the
potential for substantial performance gains since the addition-based
formulation can avoid the expansion to double precision often required for
multiplication.