Deep learning is becoming more widespread in its application due to its power in solving complex classification problems. However, deep learning models often require large memory and energy consumption, which may prevent them from being deployed effectively on embedded platforms, limiting their applications. This work addresses the problem by proposing methods {\em Weight Reduction Quantisation} for compressing the memory footprint of the models, including reducing the number of weights and the number of bits to store each weight. Beside, applying with sparsity-inducing regularization, our work focuses on speeding up stochastic variance reduced gradients (SVRG) optimization on non-convex problem. Our method that mini-batch SVRG with $\ell$1 regularization on non-convex problem has faster and smoother convergence rates than SGD by using adaptive learning rates. Experimental evaluation of our approach uses MNIST and CIFAR-10 datasets on LeNet-300-100 and LeNet-5 models, showing our approach can reduce the memory requirements both in the convolutional and fully connected layers by up to 60$\times$ without affecting their test accuracy. Artificial intelligence is finding wider application across a number of domains where computational resources can vary from large data centres to mobile devices. However, state-ofthe-art techniques such as deep learning BID14 require significant resources, including large memory requirements and energy consumption. Reducing the size of the deep learning model to a compact model that has small memory footprint without compromising its performance is a desirable research aim to address the challenges for deploying these leading approaches on mobile devices. 1 regularization can be used as a penalty to train models to prevent the model from over-fitting the training data. As well as providing, 1 regularization is a powerful compression techniques to penalize some weights to be zero. As the results, our research focus on improving the method based on 1 regularization to reduce memory requirements. Moreover, as deep neural network optimization is a non-convex problem, the optimization can be stuck in local-minimal, which can reduce the performance. To address the problem, we improve SGD optimization for non-convex function to enhancing sparse representations obtained with 1 regularization. In this paper, we propose our compression method Weight Reduction Quantisation which reduces both the number of weights and bits-depth of model without sacrificing accuracy. To reduces the number of weights, our method employs sparsity-inducing 1 regularization to encourage many connections in both convolutional and fully connected layers to be zero during the training process. Formally, in this paper we consider the following unconstrained minimization problem, Given training labels y 1 , y 2 , ..., y N as correct outputs for input data x 1 , x 2 , ..., x N , the optimization problem to estimate the weights in all layers, W, is defined by DISPLAYFORM0 L(y i , f(x i ; W)) + λr(W),where λ is a hyper-parameter controlling the degree of regularization and the weights in all layers is given by W. The problem 1 can be strongly convex or possibly non-convex BID2 . Following update rule, the mini-batch SGD method with 1 regularization is a popular approach for performing the optimization, and the weight update rule is given by DISPLAYFORM1 where each weight of network can be represented by w j ,the total number of weights is M. k is the iteration counter and η k is the learning rate and B is mini-batch size (1 < B < N) used to approximate the full gradient. However, SGD optimization with 1 regularization has two challenges: firstly, it inefficiently encourages weight to be zero due to fluctuations generated by SGD BID18 . Secondly, SGD optimization slowing down convergence rate due to the high variance of gradients. The two methods of cumulative 1 regularization and SVRG can solve the two challenges respectively:Cumulative 1 regularization BID18 proposed a method cumulating the 1 penalties to resolve the problem. The method clips regularization at zero, which avoids the derivative DISPLAYFORM2 being non-differentiable when w j = 0 and provides a more stable convergence for the weights. Moreover, the cumulative penalty can reduce the weight to zero more quickly.Mini-batch SVRG As SGD optimization has slow convergence asymptotically due to noise, BID12 proposed SVRG that can efficiently decrease the noise of SGD by reducing the variance of gradients by: DISPLAYFORM3 whereμ j is the average gradient of sub-optimal weightsw j which is the weight after every m SGD iterationsμ DISPLAYFORM4 whereW is the sub-optimal weights after m SGD iterations in all layers. For succinctness we also write ∇ψ i (w DISPLAYFORM5 . They determined that reduction of variance helps initial weights w 0 close to global minima at the beginning in order to boost the convergence rate of SGD in strongly convex problems. BID12 further prove that the performance of SGD degrades with mini-batching by the theoretical result of complexity. Specifically, for batch size of B, SGD has a 1/ √ B dependence on the batch size. In contrast, SVRG in a parallel setting has 1/B dependence on the batch size which is much better than SGD. Hence, SVRG allows more efficient mini-batching. However, for non-strongly convex problems, global minimization of non-convex function is NP-hard BID1 . BID12 have a assumption that SVRG can also be applied in neural networks to accelerate the local convergence rate of SGD. Further, Allen BID1 prove non-asymptotic rates of convergence of SVRG for non-convex optimization and proposed improved SVRG that is provably faster than SGD. Hence, a promising approach is to use mini-batch SVRG instead of SGD with cumulative 1 regularization. In this paper, we proposed Weight Reduction Quantisation that efficiently compressed neural networks without scarifying accuracy. Our method has two stages that reduce the number of weights and reduce the number of bits to store each weight. We show that SVRG and cumulative 1 regularization can improve over SGD and 1-regularization. By combining them, we have presented a new compression method Delicate-SVRG-cumulative-1 that can efficiently reduce the number of parameters by the separate adaptive learning rates. The three adaptive learning rates are applied on SVRG and cumulative 1 penalty, which provides a high accuracy and reduced number of weights. Besides, our method improved SVRG that can be used on non-convex problem with fast convergence rate. In our experiments on LeNet-300-100 and LeNet-5, our method can significantly reduce the memory requirements up to 60× without accuracy loss. After compression by our method, a compact deep neural network can be efficiently deployed on an embedded device with performance of the original model.Algorithm 1 Delicate-SVRG-cumulative-1: Stochastic descent training with cumulative 1 penalty procedure Train( λ) u ← 0 µ ← 0 Initial w j and q j with zero for all number of weights M for k=0 to Maximal Iterations do DISPLAYFORM0 end for u ← u + ηλ/M end for for j ∈ features used in sample i do randomly select m features from train samples DISPLAYFORM1 (∇ψ i (w j ) -∇ψ i (w j )) + β kμ ∇ψ i (w j ) = ∇ψ i (w j ) if w j andw j converge to the same weights theñ µ = 0 end if µ ←μ + The results showed in Figure4.7.3 Using multiple initializations to compare the performance of our method and other three methods.The experiments were run with multiple initializations and there was some small variability in the results. However, the relative performance of the our method is always better than SVRG and SGD combining with cumulative 1 regularization. The results showed in (b) The test loss: MNIST dataset on LeNet-5 (left) and CIFAR-10 dataset on LeNet-300-100 (middle) and Figure 4: Estimate the convergence rate when using four compression methods, including our method Delicate-SVRG-cumulative-1, Delicate-SVRG-cumulative-1 (without BiasPruning) that without bias-based pruning in 1 regularization,SVRG-cumulative-1 and SGD-cumulative-1, on LeNet-300-100 and LeNet-5 models with MNIST and CIFAR-10 datasets. Here we choose the compression rate that equal 90% to observe training and test loss. For MNIST dataset, we did not notice subtle difference train and test loss on LeNet-300-100 model generated by four methods. Figure 5: Using three types of initial weights, we compare our method with other three methods. D-SVRG-C-L1 and D-SVRG-C-L1(wo BiasPruning) are always better than other two methods. This experiment also can verify the our view that the performance of SVRG is better or worse than SGD that depends on the number of training samples. In our experiment, if choosing small dataset (e.g. MNIST), SVRG is better than SGD. Otherwise, if choosing relatively large dataset (e.g. CIFAR-10), SVRG is worse than SGD.
Compression of Deep neural networks deployed on embedded device.  The authors present an l-1 regularized SVRG based training algorithm that is able to force many weights of the network to be 0. This work reduces memory requirements.