Hierarchical Temporal Memory using Memristor Networks: A Survey
---
paper_title: Pattern Recognition by Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory (HTM) is still largely unknown by the pattern recognition community and only a few studies have been published in the scientific literature. This paper reviews HTM architecture and related learning algorithms by using formal notation and pseudocode description. Novel approaches are then proposed to encode coincidence-group membership (fuzzy grouping) and to derive temporal groups (maxstab temporal clustering). Systematic experiments on three line-drawing datasets have been carried out to better understand HTM peculiarities and to extensively compare it against other well-know pattern recognition approaches. Our results prove the effectiveness of the new algorithms introduced and that HTM, even if still in its infancy, compares favorably with other existing technologies.
---
paper_title: Fruit Recognition by Hierarchical Temporal Memory

paper_content:

Food is a requirement for living, and traded in enormous amounts everyday. The globalization has led to optimization of the supermarkets and that a lot of stores have introduced self-scanning systems at check out and payment. When articles such as vegetables and fruits are traded the process becomes slower because the packages usually do not wear barcodes, which have to be added manually. This is a problem and the purpose with self-scanning drops out. In this thesis, a recognition system is built with the purpose to be used in self-scanning systems. The system thresholds the original image into a binary image. The binary image is sent to an advanced type of Neural Network called Hierarchical Temporal Memory. Such a network is independent of color, size, spatial space and rotations. These properties make it suitable for the given task. Two sorts of fruits were tested and the algorithm gave the accurate prediction in 97.5 % when tested on previously unseen images.
---
paper_title: Object Categorization Using VFA-generated Nodemaps and Hierarchical Temporal Memories

paper_content:

Object categorization and recognition have proved to be difficult tasks in artificial intelligence for several decades. With the recent emergence of biologically inspired soft-computing methods, promising results in specialized application domains are more and more common. In this paper, we propose a novel object categorization method based on statistical properties of nodes -derived from the VFA model -and hierarchical temporal memories. A referential categorization method, obtained by feeding grayscale pixel levels to hierarchical temporal memories, is used to evaluate the model's performance. Results show that categorization based on the statistics of nodes seems to yield higher success rates. This is in correspondence with Biederman's conjecture in his theory of recognition by components (RBC), according to which the statistics of nodes, end points and corners carry essential and sufficient information for object recognition [1]. The first section of this paper consists of a brief introduction, in which we restate the formal definition of the VFA model, as well as present its node-filtering applications. This will be followed by a presentation of the HTM theory for size-and orientation-invariant object representation. Finally, we give a detailed case study in which a hierarchical temporal memory is used to distinguish between two, as well as several object categories.
---
paper_title: Application of Numenta® Hierarchical Temporal Memory for land-use classification

paper_content:

rays are projected upon the retina, and the brain tries to make sense of the world by means of interpreting the visual input pattern. The sensitivity and specificity with which the brain solves this computationally complex problem cannot yet be replicated on a computer. The most imposing of these problems is that of invariant visual pattern recognition. Recently it has been said that the prediction of future sensory input from salient features of current input is the keystone of intelligence. 2 The neocortex is the structure in the brain which is assumed to be responsible for the evolution of intelligence. Current sensory input patterns activate stored traces of previous inputs which then generate top-down expectations, which are verified against the bottom-up input signals. If the verification succeeds, the predicted pattern is recognised. This theory explains how humans, and mammals in general, can recognise images despite changes in location, size and lighting conditions, and in the presence of deformations and large amounts of noise. Parts of this theory, known as the memory-prediction theory (MPT), are modelled in the Hierarchical Temporal Memory or HTM technology developed by a company called Numenta ® ; 3 the model is an attempt to replicate the structural and algorithmic properties of the neocortex. 3 Spatial and temporal relations between features of the sensory signals are formed in an hierarchical memory architecture during a learning process. When a new pattern arrives, the recognition process can be viewed as choosing the stored representation that best predicts the pattern. Hierarchical Temporal Memory has been successfully applied to the recognition of relatively simple images, 4
---
paper_title: Spoken Digit Recognition using a Hierarchical Temporal Memory

paper_content:

In this paper we explore the feasibility of the MemoryPrediction Theory, implemented in the form of a Hierarchical Temporal Memory (HTM), for automatic speech recognition. Up to nowHTMs have almost exclusively been applied to image processing. However, the underlying theory can also be used as an approach to active perception of audio signals. Using the software platform under development by NUMENTA we implemented a system for isolated digit recognition, the speech recognition task that can be most easily cast in a form similar to image recognition. Our results show that the HTM approach holds promises for speech recognition. At the same time it is clear that the present implementation is not ideally suited for processing signals that encode information mainly in dynamic changes.
---
paper_title: Object Categorization Using VFA-generated Nodemaps and Hierarchical Temporal Memories

paper_content:

Object categorization and recognition have proved to be difficult tasks in artificial intelligence for several decades. With the recent emergence of biologically inspired soft-computing methods, promising results in specialized application domains are more and more common. In this paper, we propose a novel object categorization method based on statistical properties of nodes -derived from the VFA model -and hierarchical temporal memories. A referential categorization method, obtained by feeding grayscale pixel levels to hierarchical temporal memories, is used to evaluate the model's performance. Results show that categorization based on the statistics of nodes seems to yield higher success rates. This is in correspondence with Biederman's conjecture in his theory of recognition by components (RBC), according to which the statistics of nodes, end points and corners carry essential and sufficient information for object recognition [1]. The first section of this paper consists of a brief introduction, in which we restate the formal definition of the VFA model, as well as present its node-filtering applications. This will be followed by a presentation of the HTM theory for size-and orientation-invariant object representation. Finally, we give a detailed case study in which a hierarchical temporal memory is used to distinguish between two, as well as several object categories.
---
paper_title: A hierarchical Bayesian model of invariant pattern recognition in the visual cortex

paper_content:

We describe a hierarchical model of invariant visual pattern recognition in the visual cortex. In this model, the knowledge of how patterns change when objects move is learned and encapsulated in terms of high probability sequences at each level of the hierarchy. Configuration of object parts is captured by the patterns of coincident high probability sequences. This knowledge is then encoded in a highly efficient Bayesian network structure. The learning algorithm uses a temporal stability criterion to discover object concepts and movement patterns. We show that the architecture and algorithms are biologically plausible. The large scale architecture of the system matches the large scale organization of the cortex and the micro-circuits derived from the local computations match the anatomical data on cortical circuits. The system exhibits invariance across a wide variety of transformations and is robust in the presence of noise. Moreover, the model also offers alternative explanations for various known cortical phenomena.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: Continuous Online Sequence Learning with an Unsupervised Neural Network Model

paper_content:

The ability to recognize and predict temporal sequences of sensory inputs is vital for survival in natural environments. Based on many known properties of cortical neurons, hierarchical temporal memory (HTM) sequence memory is recently proposed as a theoretical framework for sequence learning in the cortex. In this paper, we analyze properties of HTM sequence memory and apply it to sequence learning and prediction problems with streaming data. We show the model is able to continuously learn a large number of variable-order temporal sequences using an unsupervised Hebbian-like learning rule. The sparse temporal codes formed by the model can robustly handle branching temporal sequences by maintaining multiple predictions until there is sufficient disambiguating evidence. We compare the HTM sequence memory with other sequence learning algorithms, including statistical methods: autoregressive integrated moving average (ARIMA), feedforward neural networks: online sequential extreme learning machine (ELM), and recurrent neural networks: long short-term memory (LSTM) and echo-state networks (ESN), on sequence prediction problems with both artificial and real-world data. The HTM model achieves comparable accuracy to other state-of-the-art algorithms. The model also exhibits properties that are critical for sequence learning, including continuous online learning, the ability to handle multiple predictions and branching sequences with high order statistics, robustness to sensor noise and fault tolerance, and good performance without task-specific hyper- parameters tuning. Therefore the HTM sequence memory not only advances our understanding of how the brain may solve the sequence learning problem, but is also applicable to a wide range of real-world problems such as discrete and continuous sequence prediction, anomaly detection, and sequence classification.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing

paper_content:

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than $200\times$ lower energy as compared with a 45-nm CMOS ASIC design.
---
paper_title: A Mathematical Formalization of Hierarchical Temporal Memory’s Spatial Pooler

paper_content:

Hierarchical temporal memory (HTM) is an emerging machine learning algorithm, with the potential to provide a means to perform predictions on spatiotemporal data. The algorithm, inspired by the neocortex, currently does not have a comprehensive mathematical framework. This work brings together all aspects of the spatial pooler (SP), a critical learning component in HTM, under a single unifying framework. The primary learning mechanism is explored, where a maximum likelihood estimator for determining the degree of permanence update is proposed. The boosting mechanisms are studied and found to be only relevant during the initial few iterations of the network. Observations are made relating HTM to well-known algorithms such as competitive learning and attribute bagging. Methods are provided for using the SP for classification as well as dimensionality reduction. Empirical evidence verifies that given the proper parameterizations, the SP may be used for feature learning.
---
paper_title: The HTM Spatial Pooler – a neocortical algorithm for online sparse distributed coding

paper_content:

Each region in the cortex receives input through millions of axons from sensory organs and from other cortical regions. It remains a mystery how cortical neurons learn to form specific connections from this large number of unlabeled inputs in order to support further computations. Hierarchical temporal memory (HTM) provides a theoretical framework for understanding the computational principles in the neocortex. In this paper we describe an important component of HTM, the HTM spatial pooler that models how neurons learn feedforward connections. The spatial pooler converts arbitrary binary input patterns into sparse distributed representations (SDRs) using competitive Hebbian learning rules and homeostasis excitability control mechanisms. Through a series of simulations, we demonstrate the key computational properties of HTM spatial pooler, including preserving semantic similarity among inputs, fast adaptation to changing statistics of the inputs, improved noise robustness over learning, efficient use of all cells and flexibility in the event of cell death or loss of input afferents. To quantify these properties, we developed a set of metrics that can be directly measured from the spatial pooler outputs. These metrics can be used as complementary performance indicators for any sparse coding algorithm. We discuss the relationship with neuroscience and previous studies of sparse coding and competitive learning. The HTM spatial pooler represents a neurally inspired algorithm for learning SDRs from noisy data streams online.
---
paper_title: Non-volatile Hierarchical Temporal Memory: Hardware for Spatial Pooling

paper_content:

Hierarchical Temporal Memory (HTM) is a biomimetic machine learning algorithm imbibing the structural and algorithmic properties of the neocortex. Two main functional components of HTM that enable spatio-temporal processing are the spatial pooler and temporal memory. In this research, we explore a scalable hardware realization of the spatial pooler closely coupled with the mathematical formulation of spatial pooler. This class of neuromorphic algorithms are advantageous in solving a subset of the future engineering problems by extracting nonintuitive patterns in complex data. The proposed architecture, Non-volatile HTM (NVHTM), leverages large-scale solid state flash memory to realize a optimal memory organization, area and power envelope. A behavioral model of NVHTM is evaluated against the MNIST dataset, yielding 91.98% classification accuracy. A full custom layout is developed to validate the design in a TSMC 180nm process. The area and power profile of the spatial pooler are 30.538mm2 and 64.394mW, respectively. This design is a proof-of-concept that storage processing is a viable platform for large scale HTM network models.
---
paper_title: SPICE modeling of memristors

paper_content:

In this paper we present a SPICE model for the titanium dioxide memristor device from its modeling equations as described in [1] and compare the SPICE simulations to the experimental data.
---
paper_title: A read-monitored write circuit for 1T1M multi-level memristor memories

paper_content:

Technology migration into nano and molecular scales has led to the design of several hybrid CMOS/nano logic and memory architectures that aim to achieve high device density with low power consumption. The discovery of the memristor has further enabled the realization of denser nanoscale memory and logic systems by facilitating the implementation of multi-level logic. In this work we propose a sneak-path free memory architecture, the 1T1M (1 transistor per memristor) that provides for 2-bit storage in each data cell (memristor). Robust read and write methodologies for the proposed architecture are also discussed and tradeoffs between faster write speeds and larger read noise margins are also analyzed. Another highlight of this work is the usage of the exponential drift memristor model to further enhance write speeds of these devices which are otherwise much slower.
---
paper_title: Two-Step Write Scheme for Reducing Sneak-Path Leakage in Complementary Memristor Array

paper_content:

In this paper, a new two-step write scheme is proposed to minimize sneak-path leakage in complementary memristor (CM) array, where no selection device is needed. When R RESET/R SET = 100, the new two-step write scheme can increase the array size of CMs 10 times larger than the conventional write. If R RESET/R SET is increased to 500, we can increase the passive array size up to 1000 × 1000 with maintaining the read sensing margin lager than 10% of VDD. The two-step write scheme will be very essential in realizing passive cross-point array without any selection device that is known to be the ideal architecture for future 3-D memories.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: Memristor SPICE model and crossbar simulation based on devices with nanosecond switching time

paper_content:

This paper presents a memristor SPICE model that is able to reproduce current-voltage relationships of previously published memristor devices. This SPICE model shows a stronger correlation to various published device data when compared to existing SPICE models. Furthermore, switching characteristics of published memristor devices with switching times in the nanosecond scale were modeled. Therefore, this model can be used to accurately simulate neural systems based on these high-speed memristors. This paper also demonstrates how this model can be used to accurately calculate switching energy of these high-speed devices, leading to more accurate power calculations in memristor based neural systems. Memristor crossbar circuits provide a potential method for developing very high density neural classifiers. This model was able to simulate crossbar circuits containing up to 256 memristors. It is significantly less likely to cause convergence errors when operating in the nanosecond switching regime with a large number of devices when compared with existing SPICE models.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: Memristor models for SPICE simulation of extremely large memristive networks

paper_content:

In order to avoid non-convergence and other numerical problems associated with the simulation of extremely large memristive networks, suitable models of the individual memristors need to be developed. A circuit containing thousands of memristors for finding the shortest path in a complicated maze is a typical example of a numerically challenging simulation. A benchmark circuit for testing the applications of various complexities is used for the transient analysis. It is shown that various commonly used models, from simple behavioral to Pickett models, require special modifications for minimizing the simulation time and improving the convergence issue.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing

paper_content:

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than $200\times$ lower energy as compared with a 45-nm CMOS ASIC design.
---
paper_title: The Art of Finding Accurate Memristor Model Solutions

paper_content:

One of the main issues preventing a large-scale exploration of the full potential of memristors in electrical circuits lies in the convergence issues and numerical errors encountered in the computer-aided integration of the differential algebraic equation set governing the peculiar dynamical behavior of these nonlinear two-terminal electrical components. In most cases the complexity of this equation set prevents an analytical derivation of closed-form state solutions. Therefore the investigation of the nonlinear dynamics of memristors and circuits based upon them relies on software-based integration of the mathematical equations. In this paper, we highlight solution accuracy issues which may arise from an improper numerical integration of the equations, and then propose techniques addressing the problems properly. These guidelines represent a useful guide to engineers interested in the numerical analysis of memristor models.
---
paper_title: A Survey of Memristive Threshold Logic Circuits

paper_content:

In this paper, we review different memristive threshold logic (MTL) circuits that are inspired from the synaptic action of the flow of neurotransmitters in the biological brain. The brainlike generalization ability and the area minimization of these threshold logic circuits aim toward crossing Moore’s law boundaries at device, circuits, and systems levels. Fast switching memory, signal processing, control systems, programmable logic, image processing, reconfigurable computing, and pattern recognition are identified as some of the potential applications of MTL systems. The physical realization of nanoscale devices with memristive behavior from materials, such as TiO2, ferroelectrics, silicon, and polymers, has accelerated research effort in these application areas, inspiring the scientific community to pursue the design of high-speed, low-cost, low-power, and high-density neuromorphic architectures.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: Feature extraction without learning in an analog Spatial Pooler memristive-CMOS circuit design of Hierarchical Temporal Memory

paper_content:

Hierarchical temporal memory (HTM) is a neuromorphic algorithm that emulates sparsity, hierarchy and modularity resembling the working principles of neocortex. Feature encoding is an important step to create sparse binary patterns. This sparsity is introduced by the binary weights and random weight assignment in the initialization stage of the HTM. We propose the alternative deterministic method for the HTM initialization stage, which connects the HTM weights to the input data and preserves natural sparsity of the input information. Further, we introduce the hardware implementation of the deterministic approach and compare it to the traditional HTM and existing hardware implementation. We test the proposed approach on the face recognition problem and show that it outperforms the conventional HTM approach.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Emerging Memristor-Based Logic Circuit Design Approaches: A Review

paper_content:

This article is a comprehensive review of the state-of-theart of memristor-based logic circuit design concepts of the recent literature. Amongst all the identified circuit design approaches, those discussed here are all based on collective memristive dynamics and share a number of common characteristics which facilitate their comparison. The focus is on the evolution of the memristor-based logic circuit design strategies from the early proposed sequential stateful logic up to most recently published design schemes which support parallel processing of the applied input signals. The main operational properties of all the selected computational concepts are presented in an accessible manner, aiming to serve as an informative cornerstone for students and scientists who wish to get involved in emerging memristive logic circuit research and development.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: Feature extraction without learning in an analog Spatial Pooler memristive-CMOS circuit design of Hierarchical Temporal Memory

paper_content:

Hierarchical temporal memory (HTM) is a neuromorphic algorithm that emulates sparsity, hierarchy and modularity resembling the working principles of neocortex. Feature encoding is an important step to create sparse binary patterns. This sparsity is introduced by the binary weights and random weight assignment in the initialization stage of the HTM. We propose the alternative deterministic method for the HTM initialization stage, which connects the HTM weights to the input data and preserves natural sparsity of the input information. Further, we introduce the hardware implementation of the deterministic approach and compare it to the traditional HTM and existing hardware implementation. We test the proposed approach on the face recognition problem and show that it outperforms the conventional HTM approach.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing

paper_content:

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than $200\times$ lower energy as compared with a 45-nm CMOS ASIC design.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Approach of facial surfaces by contour

paper_content:

In this work we have made a face recognition system based on the analysis of facial surfaces by analyzing the contours extracted from the 3D representation of human faces, called facial curves. Our recognition system uses the face representation of the image to the gray level as 2D surface in 3D space, with the third coordinate proportional to the intensity values of pixels. To extract the feature vector of face we used the method of analysis of facial curves. This approach is based on the mathematical definition of the space of facial surfaces. In this space the deformations are modeled by the geodesic path between two facial surfaces. The proposed method has been used for face recognition and tested on the database of face images known ORL database, developed by AT & T Laboratories Cambridge University in England, was used to evaluate the performance of our method under conditions where the pose and sample size are varied.
---
paper_title: A detailed survey on 2D and 3D still face and face video databases part I

paper_content:

This paper describes in detail about 2D and 3D face and video face databases developed in the year from 1992 to 2009. The still face database discussed here includes: Our Laboratory Database of Faces (ORL), Yale face database, Yale face databaseB(B stands for Big), Yale face database extendedB, FERET(Face Recognition Technology) face database and CASPEAL is a large scale Chinese face database with varying PEAL(Pose Expression Accessory, and Lighting). The 3D face databases like MIT-CBCL (Centre for Biological and Computational Learning) face recognition database, Georgia Tech face database, Cohn-Kanade Action Unit Face database and 3D BFM (Basel Face Model) are also discussed in this paper. In CK AU, CK+AU(Cohn-Kanade Action Unit) coded facial expression database, a 3D model is developed from a 2D model which is obtained from the captured video of different subjects. The other Face databases and development of new Face databases are discussed at the end of this paper. The application of proposed concept is to develop a new small size face database which includes all the important criteria included in the previous face and video face databases.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: Hierarchical Temporal Memory Based on Spin-Neurons and Resistive Memory for Energy-Efficient Brain-Inspired Computing

paper_content:

Hierarchical temporal memory (HTM) tries to mimic the computing in cerebral neocortex. It identifies spatial and temporal patterns in the input for making inferences. This may require a large number of computationally expensive tasks, such as dot product evaluations. Nanodevices that can provide direct mapping for such primitives are of great interest. In this paper, we propose that the computing blocks for HTM can be mapped using low-voltage, magnetometallic spin-neurons combined with an emerging resistive crossbar network, which involves a comprehensive design at algorithm, architecture, circuit, and device levels. Simulation results show the possibility of more than $200\times$ lower energy as compared with a 45-nm CMOS ASIC design.
---
paper_title: Non-volatile Hierarchical Temporal Memory: Hardware for Spatial Pooling

paper_content:

Hierarchical Temporal Memory (HTM) is a biomimetic machine learning algorithm imbibing the structural and algorithmic properties of the neocortex. Two main functional components of HTM that enable spatio-temporal processing are the spatial pooler and temporal memory. In this research, we explore a scalable hardware realization of the spatial pooler closely coupled with the mathematical formulation of spatial pooler. This class of neuromorphic algorithms are advantageous in solving a subset of the future engineering problems by extracting nonintuitive patterns in complex data. The proposed architecture, Non-volatile HTM (NVHTM), leverages large-scale solid state flash memory to realize a optimal memory organization, area and power envelope. A behavioral model of NVHTM is evaluated against the MNIST dataset, yielding 91.98% classification accuracy. A full custom layout is developed to validate the design in a TSMC 180nm process. The area and power profile of the spatial pooler are 30.538mm2 and 64.394mW, respectively. This design is a proof-of-concept that storage processing is a viable platform for large scale HTM network models.
---
paper_title: Unconstrained Facial Images: Database for Face Recognition Under Real-World Conditions

paper_content:

The objective of this paper is to introduce a novel face database. It is composed of face images taken in real-world conditions and is freely available for research purposes at http://ufi.kiv.zcu.cz. We have created this dataset in order to facilitate to researchers a straightforward comparison and evaluation of their face recognition approaches under “very difficult” conditions. It is composed of two partitions. The first one, called Cropped images, contains automatically detected faces from photographs. The number of individuals is 605. These images are cropped and resized to have approximately the same face size. Images in the second partition, called Large images, contain not only faces, however some background objects are also present. Therefore, it is necessary to include the face detection task before the face recognition itself. This partition contains images of 530 individuals. Another contribution of this paper is to show the recognition accuracy of several state-of-the-art face recognition approaches on this dataset to provide a baseline score for further research.
---
paper_title: Evaluation of Hierarchical Temporal Memory for a Real World Application

paper_content:

A large number of real world applications, such as user support systems, can still not be performed easily by conventional algorithms in comparison with the human brain. Such intelligence is often implemented, by using probability based systems. This paper focuses on comparing the implementation of a cellular phone intention estimation example on a Bayesian Network and Hierarchical Temporal Memory. It is found that Hierarchical Temporal Memory is a system that requires little effort for designing the application, and with some extra effort, further optimised results can easily be obtained.
---
paper_title: Non-volatile Hierarchical Temporal Memory: Hardware for Spatial Pooling

paper_content:

Hierarchical Temporal Memory (HTM) is a biomimetic machine learning algorithm imbibing the structural and algorithmic properties of the neocortex. Two main functional components of HTM that enable spatio-temporal processing are the spatial pooler and temporal memory. In this research, we explore a scalable hardware realization of the spatial pooler closely coupled with the mathematical formulation of spatial pooler. This class of neuromorphic algorithms are advantageous in solving a subset of the future engineering problems by extracting nonintuitive patterns in complex data. The proposed architecture, Non-volatile HTM (NVHTM), leverages large-scale solid state flash memory to realize a optimal memory organization, area and power envelope. A behavioral model of NVHTM is evaluated against the MNIST dataset, yielding 91.98% classification accuracy. A full custom layout is developed to validate the design in a TSMC 180nm process. The area and power profile of the spatial pooler are 30.538mm2 and 64.394mW, respectively. This design is a proof-of-concept that storage processing is a viable platform for large scale HTM network models.
---
paper_title: Performance of a hierarchical temporal memory network in noisy sequence learning

paper_content:

As neurobiological evidence points to the neocortex as the brain region mainly involved in high-level cognitive functions, an innovative model of neocortical information processing has been recently proposed. Based on a simplified model of a neocortical neuron, and inspired by experimental evidence of neocortical organisation, the Hierarchical Temporal Memory (HTM) model attempts at understanding intelligence, but also at building learning machines. This paper focuses on analysing HTM's ability for online, adaptive learning of sequences. In particular, we seek to determine whether the approach is robust to noise in its inputs, and to compare and contrast its performance and attributes to an alternative Hidden Markov Model (HMM) approach. We reproduce a version of a HTM network and apply it to a visual pattern recognition task under various learning conditions. Our first set of experiments explore the HTM network's capability to learn repetitive patterns and sequences of patterns within random data streams. Further experimentation involves assessing the network's learning performance in terms of inference and prediction under different noise conditions. HTM results are compared with those of a HMM trained at the same tasks. Online learning performance results demonstrate the HTM's capacity to make use of context in order to generate stronger predictions, whereas results on robustness to noise reveal an ability to deal with noisy environments. Our comparisons also, however, emphasise a manner in which HTM differs significantly from HMM, which is that HTM generates predicted observations rather than hidden states, and each observation is a sparse distributed representation.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: Feature extraction without learning in an analog Spatial Pooler memristive-CMOS circuit design of Hierarchical Temporal Memory

paper_content:

Hierarchical temporal memory (HTM) is a neuromorphic algorithm that emulates sparsity, hierarchy and modularity resembling the working principles of neocortex. Feature encoding is an important step to create sparse binary patterns. This sparsity is introduced by the binary weights and random weight assignment in the initialization stage of the HTM. We propose the alternative deterministic method for the HTM initialization stage, which connects the HTM weights to the input data and preserves natural sparsity of the input information. Further, we introduce the hardware implementation of the deterministic approach and compare it to the traditional HTM and existing hardware implementation. We test the proposed approach on the face recognition problem and show that it outperforms the conventional HTM approach.
---
paper_title: A design of HTM spatial pooler for face recognition using memristor-CMOS hybrid circuits

paper_content:

Hierarchical Temporal Memory (HTM) is a machine learning algorithm that is inspired from the working principles of the neocortex, capable of learning, inference, and prediction for bit-encoded inputs. Spatial pooler is an integral part of HTM that is capable of learning and classifying visual data such as objects in images. In this paper, we propose a memristor-CMOS circuit design of spatial pooler and exploit memristors capabilities for emulating the synapses, where the strength of the weights is represented by the state of the memristor. The proposed design is validated on a challenging application of single image per person face recognition problem using AR database resulting in a recognition accuracy of 80%.
---
paper_title: HTM Spatial Pooler With Memristor Crossbar Circuits for Sparse Biometric Recognition

paper_content:

Hierarchical Temporal Memory (HTM) is an online machine learning algorithm that emulates the neo-cortex. The development of a scalable on-chip HTM architecture is an open research area. The two core substructures of HTM are spatial pooler and temporal memory. In this work, we propose a new Spatial Pooler circuit design with parallel memristive crossbar arrays for the 2D columns. The proposed design was validated on two different benchmark datasets, face recognition, and speech recognition. The circuits are simulated and analyzed using a practical memristor device model and 0.18 $\mu$ m IBM CMOS technology model. The databases AR, YALE, ORL, and UFI, are used to test the performance of the design in face recognition. TIMIT dataset is used for the speech recognition.
---
paper_title: On-chip Face Recognition System Design with Memristive Hierarchical Temporal Memory

paper_content:

Hierarchical Temporal Memory is a new machine learning algorithm intended to mimic the working principle of neocortex, part of the human brain, which is responsible for learning, classification, and making predictions. Although many works illustrate its effectiveness as a software algorithm, hardware design for HTM remains an open research problem. Hence, this work proposes an architecture for HTM Spatial Pooler and Temporal Memory with learning mechanism, which creates a single image for each class based on important and unimportant features of all images in the training set. In turn, the reduction in the number of templates within database reduces the memory requirements and increases the processing speed. Moreover, face recognition analysis indicates that for a large number of training images, the proposed design provides higher accuracy results (83.5\%) compared to only Spatial Pooler design presented in the previous works.
---
paper_title: ReRAM Crossbar based Recurrent Neural Network for human activity detection

paper_content:

We present a programmable high-efficient Recurrent Neural Network (RNN) with Synapses design using Resistive Random Access Memory (ReRAM). The presented ReRAM-RNN employs crossbar ReRAM arrays as synapses. A fast synapses programming methodology is realized by CMOS-based neuron with in-built programming circuitry. The simulations are performed using experimentally verified physical resistive switching model, instead of only functional models, providing better estimate of system speed and power efficiency. Simulation results show that ReRAM-RNN can provide higher computation efficiency and/or more compact design than software realization of RNN, and dedicated CMOS based digital-and analog-RNN. We show that the efficiency improvement of ReRAM-based neural network design is more significant in feedback networks than in feedforward networks.
---
paper_title: Memristors Empower Spiking Neurons With Stochasticity

paper_content:

Recent theoretical studies have shown that probabilistic spiking can be interpreted as learning and inference in cortical microcircuits. This interpretation creates new opportunities for building neuromorphic systems driven by probabilistic learning algorithms. However, such systems must have two crucial features: 1) the neurons should follow a specific behavioral model, and 2) stochastic spiking should be implemented efficiently for it to be scalable. This paper proposes a memristor-based stochastically spiking neuron that fulfills these requirements. First, the analytical model of the memristor is enhanced so it can capture the behavioral stochasticity consistent with experimentally observed phenomena. The switching behavior of the memristor model is demonstrated to be akin to the firing of the stochastic spike response neuron model, the primary building block for probabilistic algorithms in spiking neural networks. Furthermore, the paper proposes a neural soma circuit that utilizes the intrinsic nondeterminism of memristive switching for efficient spike generation. The simulations and analysis of the behavior of a single stochastic neuron and a winner-take-all network built of such neurons and trained on handwritten digits confirm that the circuit can be used for building probabilistic sampling and pattern adaptation machinery in spiking networks. The findings constitute an important step towards scalable and efficient probabilistic neuromorphic platforms.
---
paper_title: Leakage analysis of crossbar memristor arrays

paper_content:

Crossbar memristor arrays provide a promising high density alternative for the current memory and storage technologies. These arrays suffer from parasitic current components that significantly increase the power consumption, and could ruin the readout operation. In this work we study the trade-off between the crossbar array density and the power consumption required for its readout. Our analysis is based on simulating full memristor arrays on a SPICE platform.
---
paper_title: Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

paper_content:

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.
---
paper_title: Continuous Online Sequence Learning with an Unsupervised Neural Network Model

paper_content:

The ability to recognize and predict temporal sequences of sensory inputs is vital for survival in natural environments. Based on many known properties of cortical neurons, hierarchical temporal memory (HTM) sequence memory is recently proposed as a theoretical framework for sequence learning in the cortex. In this paper, we analyze properties of HTM sequence memory and apply it to sequence learning and prediction problems with streaming data. We show the model is able to continuously learn a large number of variable-order temporal sequences using an unsupervised Hebbian-like learning rule. The sparse temporal codes formed by the model can robustly handle branching temporal sequences by maintaining multiple predictions until there is sufficient disambiguating evidence. We compare the HTM sequence memory with other sequence learning algorithms, including statistical methods: autoregressive integrated moving average (ARIMA), feedforward neural networks: online sequential extreme learning machine (ELM), and recurrent neural networks: long short-term memory (LSTM) and echo-state networks (ESN), on sequence prediction problems with both artificial and real-world data. The HTM model achieves comparable accuracy to other state-of-the-art algorithms. The model also exhibits properties that are critical for sequence learning, including continuous online learning, the ability to handle multiple predictions and branching sequences with high order statistics, robustness to sensor noise and fault tolerance, and good performance without task-specific hyper- parameters tuning. Therefore the HTM sequence memory not only advances our understanding of how the brain may solve the sequence learning problem, but is also applicable to a wide range of real-world problems such as discrete and continuous sequence prediction, anomaly detection, and sequence classification.
---
paper_title: Demonstration of Synaptic Behaviors and Resistive Switching Characterizations by Proton Exchange Reactions in Silicon Oxide

paper_content:

Demonstration of Synaptic Behaviors and Resistive Switching Characterizations by Proton Exchange Reactions in Silicon Oxide
---
paper_title: The missing memristor found

paper_content:

Nature 453, 80–83 (2008) In Fig. 2a of this Letter, the resistance for the right-hand element of the bottom circuit should be R OFF(1 - w/D), instead of the shown value R OFF w/D. Also, the correct value of window function should be w(D - W)/D 2, rather than w(1 - W)/D 2. All simulation results presented in the paper use the correct formulae.
---
paper_title: Analog Synaptic Behavior of a Silicon Nitride Memristor.

paper_content:

In this paper, we present a synapse function using analog resistive-switching behaviors in a SiNx-based memristor with a complementary metal-oxide-semiconductor compatibility and expandability to three-dimensional crossbar array architecture. A progressive conductance change is attainable as a result of the gradual growth and dissolution of the conducting path, and the series resistance of the AlOy layer in the Ni/SiNx/AlOy/TiN memristor device enhances analog switching performance by reducing current overshoot. A continuous and smooth gradual reset switching transition can be observed with a compliance current limit (>100 μA), and is highly suitable for demonstrating synaptic characteristics. Long-term potentiation and long-term depression are obtained by means of identical pulse responses. Moreover, symmetric and linear synaptic behaviors are significantly improved by optimizing pulse response conditions, which is verified by a neural network simulation. Finally, we display the spike-timing-dependent plasticity with the multipulse scheme. This work provides a possible way to mimic biological synapse function for energy-efficient neuromorphic systems by using a conventional passive SiNx layer as an active dielectric.
---
paper_title: Nanoelectronic Programmable Synapses Based on Phase Change Materials for Brain-Inspired Computing

paper_content:

Brain-inspired computing is an emerging field, which aims to extend the capabilities of information technology beyond digital logic. A compact nanoscale device, emulating biological synapses, is needed as the building block for brain-like computational systems. Here, we report a new nanoscale electronic synapse based on technologically mature phase change materials employed in optical data storage and nonvolatile memory applications. We utilize continuous resistance transitions in phase change materials to mimic the analog nature of biological synapses, enabling the implementation of a synaptic learning rule. We demonstrate different forms of spike-timing-dependent plasticity using the same nanoscale synapse with picojoule level energy consumption.
---

Title: Hierarchical Temporal Memory using Memristor Networks: A Survey

Section 1: INTRODUCTION
Description 1: Provide a brief introduction to Hierarchical Temporal Memory (HTM) and Memristor Networks, including their relevance and importance in neuromorphic computing and the motivation behind the survey.

Section 2: HTM Overview
Description 2: Describe the Hierarchical Temporal Memory (HTM) algorithm, its inspiration from the neocortex, and the key components such as the Spatial Pooler (SP) and Temporal Memory (TM).

Section 3: HTM Theory and Mathematical Framework
Description 3: Provide a detailed explanation of the HTM algorithm, including the mathematical framework for the Spatial Pooler (SP) and Temporal Memory (TM) algorithms.

Section 4: Review of Hardware Implementations of HTM
Description 4: Discuss various hardware implementations of HTM, focusing on mixed-signal and purely analog implementations based on memristive circuits, including detailed designs and operation.

Section 5: Applications of Memristive HTM
Description 5: Survey the applications of memristive HTM, including HTM performance, power, and area calculations. Include an overview of HTM system-level implementation processes and specific applications such as face recognition and speech recognition.

Section 6: Advantages, Limitations, and Open Problems
Description 6: Provide an explicit discussion of the advantages of memristive HTM hardware, limitations of the current hardware designs, and the open problems or challenges that need to be addressed in future research.

Section 7: CONCLUSION
Description 7: Summarize the key findings of the survey, reiterate the advantages of memristive HTM implementations, and discuss the potential future directions and necessary advancements in this field.