Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge.   One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments. Deep Neural Networks have been the main driving force behind recent advancement in machine learning, notably in computer vision applications. While deep learning has become the standard approach for many tasks, performing inference on low power and constrained memory hardware is still challenging. This is especially challenging in autonomous driving in electric vehicles where high precision and high throughput constraints are added on top of the low power requirements.One approach for tackling this challenge is by training networks with binary {±1} or ternary {−1, 0, 1} weights BID0 BID15 that require an order of magnitude less memory and no multiplications, leading to significantly faster inference on dedicated hardware. The problem arises when trying to backpropagate errors as the weights are discrete. One heuristic suggested in BID0 is to use stochastic weights w, sample binary weights w b according to w for the forward pass and gradient computation and then update the stochastic weights w instead. Another idea, used by BID5 and BID15 , is to apply a "straight-through" estimator ∂sign ∂r = r1[|r| ≤ 1]. While these ideas were able to produce good results, even on reasonably large networks such as ResNet-18 BID4 , there is still a large gap in prediction accuracy between the full-precision network and the discrete networks.In this paper, we attempt to train neural networks with discrete weights using a more principled approach. Instead of trying to find a good "derivative" to a non-continuous function, we show how we can find a good smooth approximation and use its derivative to train the network. This is based on the simple observation that if at layer l we have stochastic (independent) weights w (CLT) . This allows us to model the pre-activation using a smooth distribution and use the reparameterization trick BID9 to compute derivatives. The idea of mod-eling the distribution of pre-activation, instead of the distribution of weights, was used in for Gaussian weight distributions where it was called the local reparameterization trick. We show here that with small modifications it can be used to train discrete weights and not just continuous Gaussian distributions. DISPLAYFORM0 Figure 1: The top histogram in each subfigure shows the pre-activation of a random neuron, z l i , which is calculated in a regular feed-forward setting when explicitly sampling the weights. The bottom shows samples from the approximated pre-activation using Lyapunov CLT. (a) refers to the first hidden layer whereas (b) refers to the last. We can see the approximation is very close to the actual pre-activation when sampling weights and performing standard feed-forward. In addition, we see it even holds for the first hidden layer, where the number of elements is not large (in this example, 27 elements for a 3 × 3 × 3 convolution).We experimented with both binary and ternary weights, and while the results on some datasets are quite similar, ternary weights were considerably easier to train. From a modeling perspective restricting weights to binary values {±1} forces each neuron to affect all neurons in the subsequent layer making it hard to learn representations that need to capture several independent features.In this work, we present a novel and simple method for training neural networks with discrete weights. We show experimentally that we can train binary and ternary networks to achieve stateof-the-art results on several datasets, including ResNet-18 on ImageNet, compared to previously proposed binary or ternary training algorithms. On MNIST and CIFAR-10 we can also almost match the performance of the original full-precision network using discrete weights. In this work we presented a simple, novel and effective algorithm to train neural networks with discrete weights. We showed results on various image classification datasets and reached state-ofthe-art results in both the binary and ternary settings on most datasets, paving the way for easier and more efficient training and inference of efficient low-power consuming neural networks.Moreover, comparing binary and ternary networks we advocate further research into ternary weights as a much more reasonable model than binary weights, with a modest computation and memory overhead. Further work into sparse ternary networks might help reduce, or even eliminate, this overhead compared to binary networks.
Training binary/ternary networks using local reparameterization with the CLT approximation Trains binary and ternary weight distribution networks using backpropagation to sample neuron pre-activations with reparameterization trick This paper suggests using stochastic parameters in combination with the local reparametrisation trick to train neural networks with binary or ternary weights, which leads to state of the art results.