---
author:
- 'Akul Malhotra, Sen Lu, Kezhou Yang, and Abhronil Sengupta, [^1][^2]'
title: Exploiting Oxide Based Resistive RAM Variability for Bayesian Neural Network Hardware Design
---

Neuromorphic Computing, Bayesian Neural Networks, Resistive Random Access Memory.

Introduction
============

While Bayesian deep learning has shown promise to serve as a pathway for enabling Probabilistic Machine Learning, the algorithms have been primarily developed without any insights regarding the underlying hardware implementation. Bayesian techniques are more computationally expensive than their non-Bayesian counterparts, thereby limiting their training and deployment in resource-constrained environments like wearables and mobile edge devices. In addition to the standard von-Neumann bottleneck [@zidan2018future] prevalent in current deep learning networks (where memory access and memory leakage can account for significant portion of the total energy consumption profile), Bayesian deep learning involves repeated sampling of network weights from learnt probability distributions (in most cases, Gaussian distributions are used which are much more hardware expensive than uniform probability distributions) and inference based on the sampled weights. For instance, implementing just a single synapse would involve a costly CMOS Gaussian random number generator circuit. Repeated parameter sampling and evaluation for just a single inference operation worsens the von-Neumann bottleneck issue further. With deep networks involving more than a million parameters coupled with the necessity of performing mathematical operations on sampled probabilistic data, the projected hardware costs for a typical CMOS implementation would be humongous. Thus there is a dire need to rethink hardware designs for such probabilistic machine learning frameworks ground-up where the core hardware units are better matched to the models of computation. Non-idealities and stochasticity prevalent in RRAM technologies have been typically not exploited for computing [@dalgaty2019hybrid]. While there have been recent efforts at incorporating the RRAM stochastic switching processes for various applications [@suri2015neuromorphic; @sahay2017oxram; @ly2018role; @chen2015comprehensive; @wei2016true], the concept of utilizing probability distributions obtained from RRAM cycle-to-cycle variations for Bayesian deep learning applications have been not explored before.

Utilizing Cycle-to-Cycle RRAM Variability for Probability Distribution Sampling
===============================================================================

![(a) A 1T-1R structure where the magnitude and sign of $V_{Reset/Set}$ controls the device resistance. (b) $I-V$ characteristics of our RRAM model based on parameters reported in Ref. [@jiang2016compact] for a bilayer $TiO_{x}/HfO_{x}$ material stack with $3.3nm$ thick $HfO_{x}$ layer. (c) Probability distribution of the cycle-to-cycle variation of the RESET resistance. The device is first SET using 1.1V and then RESET using -3V. Each cycle is $30ns$ in duration. The histogram represents the sampled RRAM conductance value at the end of the $30ns$ read phase. 1,000 datapoints have been used to plot the probability distribution.[]{data-label="fig1"}](fig1.pdf){width="48.00000%"}

Metal oxide RRAMs are emerging as an alternative to traditional CMOS based memories in non-Boolean computing due to their high density, CMOS compatibility and low power consumption. A metal oxide RRAM is a two-terminal device whose resistance can be changed upon applying a voltage across the terminals (see Fig. \[fig1\]). The device consists of an oxide layer sandwiched between two electrode layers. The metal oxide RRAM has two extreme resistance states, the high resistance state (HRS) and the low resistance state (LRS). When the resistance of the RRAM switches from HRS to LRS, the event is called a SET process and when the resistance switches from LRS to HRS, the event is called a RESET process. The SET and RESET processes can be explained by the growth and rupture of a conduction filament in the oxide. We used an experimentally calibrated publicly accessible RRAM model [@nanoHUB.org19] and our simulation parameters correspond to a bilayer $TiO_{x}/HfO_{x}$ material stack with $3.3nm$ thick $HfO_{x}$ layer. Please refer to Ref. [@jiang2016compact] for a description of the device model and fitting parameters. The RRAM RESET process is a gradual one and exhibits cycle-to-cycle resistance variation as depicted in Fig. \[fig1\](c). This is due to the random oxygen-vacancy filament dissolution process. While prior works have reported similar variation studies, they have typically considered this as a non-ideality. In this article, we utilize such a probability distribution sampling function for Bayesian network implementation. It is worth noting here that the distribution is not strictly Gaussian (log-normal is a better fit [@dalgaty2019hybrid; @garbin2014variability]). However, we did not observe the minor deviation from the Gaussian distribution to impact the network accuracy. The network can be also inherently trained assuming log-normal probability priors to account for the device constraints. In this work, we have assumed Gaussian probability priors for training.

System Design and Results
=========================

![RRAM Based Bayesian Neural Network Implementation. The rows of the crossbar array are driven by scaled inputs where the scaling factor is a sampling from a Normal distribution. The sampling operation can be simply implemented by an RRAM device interfaced with an amplifier and Sample-Hold circuit.[]{data-label="fig2"}](fig2.pdf){width="48.00000%"}

In a Bayesian Neural Network, the synaptic weights, $\textbf{W}$, are typically characterized by Gaussian probability distributions. Once all the *posterior* distributions are learnt ($\mu$ and $\sigma$ parameters of the weight distributions) [@gal2015bayesian], the network output corresponding to input, $\textbf{x}$, should be obtained by averaging the outputs obtained by sampling weights from the *posterior* distribution of the weights, $\textbf{W}$ [@cai2018vibnn]. The output of the network, $y$, is therefore given by, $$y = \mathbb{E}_{P(\textbf{W}|D)}[f(\textbf{x,W})] \approx \mathbb{E}_{q(\textbf{W},\theta)}[f(\textbf{x,W})] \approx \frac{1}{S}\sum_{i=1}^{S}f(\textbf{x,W}^i)$$ where, $P(D|\textbf{W})$ is the *likelihood*, corresponding to the feedforward pass of the network, $f(\textbf{x,W})$ is the network mapping for input $\textbf{x}$ and weights, $\textbf{W}$. The approximation is performed over $S$ independent Monte-Carlo samples drawn from the Gaussian distribution, $q(\textbf{W, $\theta$})$, characterized by parameters, $\theta=(\mu,\sigma$), where $\mu$ and $\sigma$ represent the mean and standard deviation vectors for the probability distributions representing $P(\textbf{W}|D)$ [@ghahramani2001propagation]. We consider the Variational Inference method [@houthooft2016vime; @jordan1999introduction] for training the network in this work.

Hence, the core computation in a Bayesian network is a dot-product between an input vector, $\textbf{x}$, and samples drawn from a Gaussian probability distribution, **$N$(, )**. Considering just a single layer and neglecting the neural transfer function, $f(\textbf{x,W}^i)$ for the $j$-th neuron can be simplified as, $$f(\textbf{x,W}^i_j) = \sum_{k}x_k . N(\mu_{jk},\sigma_{jk}) = \sum_{k}x_k . (\mu_{jk}^{\prime} + \sigma_{jk}^{\prime}.N(a,b))$$ where, $k$ is the dimensionality of the input $\textbf{x}$, $N(\mu_{jk},\sigma_{jk})$ represent a particular weight sample drawn from a Normal probability distribution with mean $\mu_{jk}$ and variance $\sigma_{jk}$ and $N(a,b)$ is the probability distribution of cycle-to-cycle resistance variation with mean $a$ and variance $b$ obtained from the RRAM device utilized for probabilistic sampling (Fig. \[fig1\](c)). It can be proved by simple algebraic manipulation that, $\mu_{jk}^{\prime}=\mu_{jk}-a/b$ and $\sigma_{jk}^{\prime}=\sigma_{jk}/b$.

While the first dot-product in Eq. (2) can be easily performed in a standard RRAM crossbar array [@mehonic2019simulation], the implementation of the second term is shown in Fig. \[fig2\]. The design consists of two components - (i) **Stochastic Unit:** An RRAM device undergoes SET-RESET cycling and its resistance is sampled during a subsequent read phase. Fig. \[fig1\](c) shows the Normal variation of the sampled conductance value. We utilize a current-to-voltage converter to scale each input to the crossbar array by a factor sampled from the Normal distribution. The amplification factor of this stage is designed such that the maximum read-voltage of the crossbar array is limited to $0.3V$ (to ensure linear RRAM $I-V$ characteristics) [@mehonic2019simulation]. Note that these devices are used as binary elements and solely serve the purpose of probability distribution sampling. (ii) **Deterministic Unit:** The crossbar array stores the parameters of the array $\boldsymbol{\sigma^{\prime}}$. The RRAM devices in the crossbar are programmed to multi-level states which can be achieved by modulating the RESET pulse amplitude, duration or the number of pulses [@cheng2017time]. Each column is read sequentially by the array peripherals in order to ensure that independent random samples are being used for each array element. The current flowing through each cross-point device is scaled by the conductance of the device and due to Kirchoff’s law, all these currents get summed up along the column, thereby realizing the dot-product kernel. The analog current outputs drive interfaced Analog-to-Digital converters (ADCs) to provide output to the fan-out neurons.

A hybrid device-circuit-algorithm co-simulation framework was developed and the design was tested for a standard digit recognition problem on the MNIST dataset [@lecun1998gradient]. The neural network used in this work consisted of 2 hidden layers each with 200 neurons. The probability distributions were learnt using the ‘Bayes by Backprop’ algorithm [@blundell2015weight] implemented in a PyTorch framework [@torch]. Verilog-A RRAM device model [@nanoHUB.org19] was used with $45nm$ Predictive Technology Model [@ptm] to implement the access transistors. The algorithmic simulator was modified to implement a hardware-aware mapping incorporating probability distribution obtained from device-circuit co-simulation framework (which deviates from the ideal Gaussian distribution), limited bit-precision and programming ranges in the crossbar array elements. We considered 4-bit representation in the RRAM devices in the cross-point array and 3-bit discretization in the Analog-to-Digital converter output (interfaced with the crossbar array). Lower bit precision resulted in degradation of classification accuracy. The RRAM crossbar programmable conductance states were linearly spaced and programmed by modulating the RESET pulse amplitude in the range of $1.4-3.2V$. Recently, more than 6-bit storage has been achieved experimentally in metal-oxide bi-layers [@stathopoulos2017multibit].

While cycle-to-cycle resistance variation of the RRAM elements of the stochastic unit are utilized for computing, other forms of variation are present in the system design. Device-to-device variations of the RRAM elements of the stochastic unit can be present which will result in variation of the parameters $a$ and $b$ of the sampled Normal distribution $N(a,b)$ for each row of the crossbar array. Further, device-to-device variations and write noise can result in variation of the crossbar RRAM resistances. Using our hybrid hardware-algorithm co-simulation framework, the test accuracy of a single sampled network was $98\%$ (over a randomly sampled 100-image subset of the test set). For the variation analysis, the average classification accuracy of 10 independent Monte-carlo runs, assuming $10\%$ Gaussian noise on each of the above-mentioned parameters (each run consisting of a single sampled network over the same 100 image test-set), was evaluated to be $93\%$. Note that this accuracy degradation can be potentially reduced by exploring hardware-in-the-loop training techniques.

Our proposal of partitioning the computation as described in Eq. (2) and hybrid system design with stochastic and deterministic units allows us to perform the costly dot-product operations (one of the main computationally expensive operations in a neural network hardware) in the memory array itself, enabling us to address the issues of von-Neumann bottleneck. From a system design perspective, the energy consumption is dominated by the Gaussian random number generation process since it involves RRAM programming while the remaining components are primarily RRAM read operations and peripheral contributions. Hence, to substantiate our benefits in contrast to CMOS hardware, we first focus on the Gaussian random number generation hardware complexity. CMOS based implementations of Gaussian random number generators rely on a significant number of linear feedback circuits which are extremely hardware expensive. For instance, a recent work for a CMOS based $64$-parallel Gaussian RNG reports $1780$ registers and $528.69mW$ power consumption [@cai2018vibnn]. In stark contrast, our implementation uses simply a single RRAM device and a current-to-voltage converter to implement the probability distribution sampling. The total power consumption for our RRAM based Gaussian random number generation is $32.54mW$ ($VI$ power consumption for the stochastic unit circuit during a complete cycle RRAM operation to generate a sample from a Gaussian distribution) for a similar 64 parallel generation task. For system level energy consumption evaluation, we considered 8-bit resolution in DAC/ADC designs to achieve iso-bit length comparison with Ref. [@cai2018vibnn]. Typical peripheral energy consumption metrics for DAC, ADC and Sample and Hold circuits reported in Refs. [@ankit2019puma; @shafiee2016isaac] were included. The crossbar read-latency per column was considered to be $10ns$. The overall energy efficiency of the system was evaluated to be $106567.2$ $Images/J$, which is $2\times$ higher than the baseline CMOS implementation [@cai2018vibnn]. It is worth noting here that device level non-idealities in other technologies can be harnessed as well for such probabilistic AI hardware. This work is based on Ref. [@yang2019all] which explored the design for spintronic technologies. This work explores the design for RRAM devices which requires a significant rethinking due to different intrinsic device physics and operating voltage/current characteristics and conditions.

Summary
=======

In conclusion, we have provided a proposal for an RRAM based “In-Memory" computing primitive for Bayesian neural hardware that utilizes simple device-circuit primitives and leverages RRAM stochasticity as a computing resource, instead of viewing it as a disadvantage. Such hardware-software co-design of Bayesian neural network models can potentially lead to real-time decision making in autonomous agents in the presence of uncertainties.

[10]{} \[1\][\#1]{} url@samestyle \[2\][\#2]{} \[2\][[l@\#1=l@\#1\#2]{}]{}

M. A. Zidan, J. P. Strachan, and W. D. Lu, “The future of electronics based on memristive systems,” *Nature Electronics*, vol. 1, no. 1, p. 22, 2018.

T. Dalgaty, M. Payvand, F. Moro, D. R. Ly, F. Pebay-Peyroula, J. Casas, G. Indiveri, and E. Vianello, “Hybrid neuromorphic circuits exploiting non-conventional properties of [RRAM]{} for massively parallel local plasticity mechanisms,” *APL Materials*, vol. 7, no. 8, p. 081125, 2019.

M. Suri, V. Parmar, A. Kumar, D. Querlioz, and F. Alibart, “Neuromorphic hybrid [RRAM-CMOS RBM]{} architecture,” in *2015 15th Non-Volatile Memory Technology Symposium (NVMTS)*.1em plus 0.5em minus 0.4emIEEE, 2015, pp. 1–6.

S. Sahay, A. Kumar, V. Parmar, and M. Suri, “[OxRAM RNG circuits exploiting multiple undesirable nanoscale phenomena]{},” *IEEE Transactions on Nanotechnology*, vol. 16, no. 4, pp. 560–566, 2017.

D. R. Ly, A. Grossi, C. Fenouillet-Beranger, E. Nowak, D. Querlioz, and E. Vianello, “Role of synaptic variability in resistive memory-based spiking neural networks with unsupervised learning,” *Journal of Physics D: Applied Physics*, vol. 51, no. 44, p. 444002, 2018.

A. Chen, “Comprehensive assessment of rram-based puf for hardware security applications,” in *2015 IEEE International Electron Devices Meeting (IEDM)*.1em plus 0.5em minus 0.4emIEEE, 2015, pp. 10–7.

Z. Wei, Y. Katoh, S. Ogasahara, Y. Yoshimoto, K. Kawai, Y. Ikeda, K. Eriguchi, K. Ohmori, and S. Yoneda, “[True random number generator using current difference based on a fractional stochastic model in 40-nm embedded ReRAM]{},” in *2016 IEEE International Electron Devices Meeting (IEDM)*.1em plus 0.5em minus 0.4emIEEE, 2016, pp. 4–8.

Z. Jiang, Y. Wu, S. Yu, L. Yang, K. Song, Z. Karim, and H.-S. P. Wong, “A compact model for metal–oxide resistive random access memory with experiment verification,” *IEEE Transactions on Electron Devices*, vol. 63, no. 5, pp. 1884–1892, 2016.

Z. Jiang and H.-S. P. Wong, “[Stanford University Resistive-Switching Random Access Memory (RRAM) Verilog-A Model]{},” Oct 2014. \[Online\]. Available: <https://nanohub.org/publications/19/1>

D. Garbin, O. Bichler, E. Vianello, Q. Rafhay, C. Gamrat, L. Perniola, G. Ghibaudo, and B. DeSalvo, “Variability-tolerant convolutional neural network for pattern recognition applications based on [OxRAM]{} synapses,” in *2014 IEEE International Electron Devices Meeting*.1em plus 0.5em minus 0.4emIEEE, 2014, pp. 28–4.

Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with [Bernoulli]{} approximate variational inference,” *arXiv preprint arXiv:1506.02158*, 2015.

R. Cai, A. Ren, N. Liu, C. Ding, L. Wang, X. Qian, M. Pedram, and Y. Wang, “[VIBNN]{}: Hardware acceleration of bayesian neural networks,” in *Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems*. 1em plus 0.5em minus 0.4emACM, 2018, pp. 476–488.

Z. Ghahramani and M. J. Beal, “Propagation algorithms for variational [Bayesian]{} learning,” in *Advances in neural information processing systems*, 2001, pp. 507–513.

R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, “[VIME]{}: Variational information maximizing exploration,” in *Advances in Neural Information Processing Systems*, 2016, pp. 1109–1117.

M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction to variational methods for graphical models,” *Machine learning*, vol. 37, no. 2, pp. 183–233, 1999.

A. Mehonic, D. Joksas, W. H. Ng, M. Buckwell, and A. J. Kenyon, “Simulation of inference accuracy using realistic rram devices,” *Frontiers in Neuroscience*, vol. 13, 2019.

M. Cheng, L. Xia, Z. Zhu, Y. Cai, Y. Xie, Y. Wang, and H. Yang, “[TIME: A training-in-memory architecture for memristor-based deep neural networks]{},” in *Proceedings of the 54th Annual Design Automation Conference 2017*.1em plus 0.5em minus 0.4emACM, 2017, p. 26.

Y. LeCun, L. Bottou, Y. Bengio, P. Haffner *et al.*, “Gradient-based learning applied to document recognition,” *Proceedings of the IEEE*, vol. 86, no. 11, pp. 2278–2324, 1998.

C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural networks,” *arXiv preprint arXiv:1505.05424*, 2015.

“[Bayes by Backprop PyTorch Code]{}.” \[Online\]. Available: <https://github.com/nitarshan/bayes-by-backprop>

“[Predictive Technology Model (PTM)]{}.” \[Online\]. Available: <http://ptm.asu.edu/>

S. Stathopoulos, A. Khiat, M. Trapatseli, S. Cortese, A. Serb, I. Valov, and T. Prodromakis, “Multibit memory operation of metal-oxide bi-layer memristors,” *Scientific reports*, vol. 7, no. 1, p. 17532, 2017.

A. Ankit, I. E. Hajj, S. R. Chalamalasetti, G. Ndu, M. Foltin, R. S. Williams, P. Faraboschi, W.-m. W. Hwu, J. P. Strachan, K. Roy *et al.*, “[PUMA]{}: A programmable ultra-efficient memristor-based accelerator for machine learning inference,” in *Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems*.1em plus 0.5em minus 0.4emACM, 2019, pp. 715–731.

A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “[ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars]{},” *ACM SIGARCH Computer Architecture News*, vol. 44, no. 3, pp. 14–26, 2016.

K. Yang, A. Malhotra, S. Lu, and A. Sengupta, “All-spin bayesian neural networks,” *arXiv preprint arXiv:1911.05828*, 2019.

[^1]: Manuscript received November, 2019.

[^2]: The authors are with the School of Electrical Engineering and Computer Science, Department of Materials Science and Engineering, The Pennsylvania State University, University Park, PA 16802, USA. A. Malhotra is also affiliated with Birla Institute of Technology and Science, Pilani, Rajasthan 333031, India. E-mail: sengupta@psu.edu.
