Hardware-friendly learning algorithms for neural networks: an overview
---
paper_title: Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks

paper_content:

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms like backpropagation. Although backpropagation is efficient, its implementation in analog VLSI requires excessive computational hardware. In this paper we show that, for analog parallel implementations, the use of gradient descent with direct approximation of the gradient using weight perturbation instead of backpropagation significantly reduces hardware complexity. Gradient descent by weight perturbation eliminates the need for derivative and bidirectional circuits for on-chip learning, and access to the output states of neurons in hidden layers for off-chip learning. We also show that weight perturbation can be used to implement recurrent networks. A discrete level analog implementation showing the training of an XOR network as an example is described.
---
paper_title: Integrated pulse stream neural networks: results, issues, and pointers

paper_content:

Results from working analog VLSI implementations of two different pulse stream neural network forms are reported. The circuits are rendered relatively invariant to processing variations, and the problem of cascadability of synapses to form large systems is addressed. A strategy for interchip communication of large numbers of neural states has been implemented in silicon and results are presented. The circuits demonstrated confront many of the issues that blight massively parallel analog systems, and offer solutions. >
---
paper_title: Learning internal representations by error propagation

paper_content:

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion
---
paper_title: IV Adaptive Multilayer Optical Networks

paper_content:

Publisher Summary This chapter discusses the adaptive multilayer optical networks. The chapter describes an experimental two-layer optical neural network recently built at California Institute of Technology. The system is trained for handwritten character recognition, and the experimental results are provided in the chapter. The implementation of fully adaptive learning algorithms in such a network is discussed. The chapter describes a local learning algorithm for fully adaptive two-layer networks. The problem of hologram decay is discussed and a solution using periodic copying is described. The chapter presents a system that provides phase-locked sustainment of photorefractive holograms. The hologram copying method is a promising solution for the device dynamic range problems. As for the training of multilayer networks, large networks require very large training sets. Understanding such algorithmic issues about the training of large optical networks is the major challenge before these systems can have a practical impact.
---
paper_title: 30 years of adaptive neural networks: perceptron, Madaline, and backpropagation

paper_content:

Fundamental developments in feedforward artificial neural networks from the past thirty years are reviewed. The history, origination, operating characteristics, and basic theory of several supervised neural-network training algorithms (including the perceptron rule, the least-mean-square algorithm, three Madaline rules, and the backpropagation technique) are described. The concept underlying these iterative adaptation algorithms is the minimal disturbance principle, which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. The two principal kinds of online rules that have developed for altering the weights of a network are examined for both single-threshold elements and multielement networks. They are error-correction rules, which alter the weights of a network to correct error in the output response to the present input pattern, and gradient rules, which alter the weights of a network during each pattern presentation by gradient descent with the objective of reducing mean-square error (averaged over all training patterns). >
---
paper_title: A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

paper_content:

Typical methods for gradient descent in neural network learning involve calculation of derivatives based on a detailed knowledge of the network model. This requires extensive, time consuming calculations for each pattern presentation and high precision that makes it difficult to implement in VLSI. We present here a perturbation technique that measures, not calculates, the gradient. Since the technique uses the actual network as a measuring device, errors in modeling neuron activation and synaptic weights do not cause errors in gradient descent. The method is parallel in nature and easy to implement in VLSI. We describe the theory of such an algorithm, an analysis of its domain of applicability, some simulations using it and an outline of a hardware implementation.
---
paper_title: Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks

paper_content:

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms like backpropagation. Although backpropagation is efficient, its implementation in analog VLSI requires excessive computational hardware. In this paper we show that, for analog parallel implementations, the use of gradient descent with direct approximation of the gradient using weight perturbation instead of backpropagation significantly reduces hardware complexity. Gradient descent by weight perturbation eliminates the need for derivative and bidirectional circuits for on-chip learning, and access to the output states of neurons in hidden layers for off-chip learning. We also show that weight perturbation can be used to implement recurrent networks. A discrete level analog implementation showing the training of an XOR network as an example is described.
---
paper_title: TOTEM: a digital processor for neural networks and Reactive Tabu Search

paper_content:

The Reactive Tabu Search (RTS) algorithm permits the training of neural networks with low number of bits per weight, low computational accuracy, no local minima “trapping”, and limited sensitivity to the initial conditions. Two architectures for the implementation of Multi-Layer Perceptrons based directly on the properties of RTS are presented. They are characterized by high-speed integer computation, tunable precision, 1/0 balanced design, internal storage of the weights and compact processing units. TOTEM, a silicon implementation which uses only 8 bits per weight and 16 bits per input, is then presented. A triggering problem in High Energy Physics is used as a test-bed for the algorithm and the processor. The eqected performance is briefly compared with those of two alternative processors.
---
paper_title: Training Neural Nets with the Reactive Tabu Search

paper_content:

In this paper the task of training subsymbolic systems is considered as a combinatorial optimization problem and solved with the heuristic scheme of the reactive tabu search (RTS). An iterative optimization process based on a "modified local search" component is complemented with a meta-strategy to realize a discrete dynamical system that discourages limit cycles and the confinement of the search trajectory in a limited portion of the search space. The possible cycles are discouraged by prohibiting (i.e., making tabu) the execution of moves that reverse the ones applied in the most recent part of the search. The prohibition period is adapted in an automated way. The confinement is avoided and a proper exploration is obtained by activating a diversification strategy when too many configurations are repeated excessively often. The RTS method is applicable to nondifferentiable functions, is robust with respect to the random initialization, and effective in continuing the search after local minima. Three tests of the technique on feedforward and feedback systems are presented. >
---
paper_title: The geometrical learning of binary neural networks.

paper_content:

In this paper, the learning algorithm called expand-and-truncate learning (ETL) is proposed to train multilayer binary neural networks (BNN) with guaranteed convergence for any binary-to-binary mapping. The most significant contribution of this paper is the development of a learning algorithm for three-layer BNN which guarantees the convergence, automatically determining a required number of neurons in the hidden layer. Furthermore, the learning speed of the proposed ETL algorithm is much faster than that of backpropagation learning algorithm in a binary field. Neurons in the proposed BNN employ a hard-limiter activation function, with only integer weights and integer thresholds. Therefore, this will greatly facilitate actual hardware implementation of the proposed BNN using currently available digital VLSI technology.
---
paper_title: 30 years of adaptive neural networks: perceptron, Madaline, and backpropagation

paper_content:

Fundamental developments in feedforward artificial neural networks from the past thirty years are reviewed. The history, origination, operating characteristics, and basic theory of several supervised neural-network training algorithms (including the perceptron rule, the least-mean-square algorithm, three Madaline rules, and the backpropagation technique) are described. The concept underlying these iterative adaptation algorithms is the minimal disturbance principle, which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. The two principal kinds of online rules that have developed for altering the weights of a network are examined for both single-threshold elements and multielement networks. They are error-correction rules, which alter the weights of a network to correct error in the output response to the present input pattern, and gradient rules, which alter the weights of a network during each pattern presentation by gradient descent with the objective of reducing mean-square error (averaged over all training patterns). >
---
paper_title: Complex domain backpropagation

paper_content:

The backpropagation algorithm is extended to complex domain backpropagation (CDBP) which can be used to train neural networks for which the inputs, weights, activation functions, and outputs are complex-valued. Previous derivations of CDBP were necessarily admitting activation functions that have singularities, which is highly undesirable. In the derivation, CDBP is derived so that that it accommodates classes of suitable activation functions. One such function is found and the circuit implementation of the corresponding neuron is given. CDBP hardware circuits can be used to process sinusoidal signals all at the same frequency (phasors). >
---
paper_title: Quantization effects in digitally behaving circuit implementations of Kohonen networks

paper_content:

Implementing a neural network on a digital or mixed analog and digital chip yields the quantization of the synaptic weights dynamics. This paper addresses this topic in the case of Kohonen's self-organizing maps. We first study qualitatively how the quantization affects the convergence and the properties, and deduce from this analysis the way to choose the parameters of the network (adaptation gain and neighborhood). We show that a spatially decreasing neighborhood function is far more preferable than the usually rectangular neighborhood function, because of the weight quantization. Based on these results, an analog nonlinear network, integrated in a standard CMOS technology, and implementing this spatially decreasing neighborhood function is then presented. It can be used in a mixed analog and digital circuit implementation. >
---
paper_title: How to modify Kohonen's self-organising feature maps for an efficient digital parallel implementation

paper_content:

Two new variants of Kohonen's self-organising feature maps based on batch processing are presented in this work. The motivation is related to the need of exploiting the hardware resources of neurocomputers based on systolic arrays. Ordering and convergence to asymptotic values for 1D maps and 1D continuous input and weight spaces are proved for both variants. Finally, simulations on uniform 2D data as well as simulations on speech 12D data using 2D maps are also presented to back the theoretical results.
---
paper_title: A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

paper_content:

Typical methods for gradient descent in neural network learning involve calculation of derivatives based on a detailed knowledge of the network model. This requires extensive, time consuming calculations for each pattern presentation and high precision that makes it difficult to implement in VLSI. We present here a perturbation technique that measures, not calculates, the gradient. Since the technique uses the actual network as a measuring device, errors in modeling neuron activation and synaptic weights do not cause errors in gradient descent. The method is parallel in nature and easy to implement in VLSI. We describe the theory of such an algorithm, an analysis of its domain of applicability, some simulations using it and an outline of a hardware implementation.
---
paper_title: RA: an analog neurocomputer for the synchronous Boltzmann machine

paper_content:

The Random Analog (RA) Neurocomputer runs synchronous Boltzmann machines dedicated to real time visual pattern recognition. Its architecture achieves scalability and modularity thanks to a linear array of analog processing modules linked by a high bandwidth interconnection network, a controller, and a front end running a high level user interface. A prototype with 128 input neurons and 64 active neurons and 4.2 GCPS peak processing power is described and simulation results are given.
---
paper_title: An Optoelectronic Architecture for Multilayer Learning in a Single Photorefractive Crystal

paper_content:

We propose a simple architecture for implementing supervised neural network models optically with photorefractive technology. The architecture is very versatile: a wide range of supervised learning algorithms can be implemented including mean-field-theory, backpropagation, and Kanerva-style networks. Our architecture is based on a single crystal with spatial multiplexing rather than the more commonly used angular multiplexing. It handles hidden units and places no restrictions on connectivity. Associated with spatial multiplexing are certain physical phenomena, rescattering and beam depletion, which tend to degrade the matrix multiplications. Detailed simulations including beam absorption and grating decay show that the supervised learning algorithms (slightly modified) compensate for these degradations.
---
paper_title: Enhanced MLP Performance and Fault Tolerance Resulting from Synaptic Weight Noise During Training

paper_content:

We analyze the effects of analog noise on the synaptic arithmetic during multilayer perceptron training, by expanding the cost function to include noise-mediated terms. Predictions are made in the light of these calculations that suggest that fault tolerance, training quality and training trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The results appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implications for all applications, particularly those involving "inaccurate" analog neural VLSI. >
---
paper_title: Analog CMOS implementation of a multilayer perceptron with nonlinear synapses

paper_content:

A neurocomputer based on a high-density analog integrated circuit developed in a 3 mu m CMOS technology has been built. The 1.6 mm*2.4 mm chip contains 18 neurons and 161 synapses in three layers, and provides 16 inputs and 4 outputs. The weights are stored on storage capacitors of the synapses. A formalization of the error back-propagation algorithm which allows the use of very small nonlinear synapses is shown. The influence of offset voltages in the synapses on the circuit performance is analyzed. Some experimental results are reported and discussed. >
---
paper_title: Performance and fault-tolerance of neural networks for optimization

paper_content:

The fault-tolerance characteristics of time-continuous, recurrent artificial neural networks (ANNs) that can be used to solve optimization problems are investigated. The performance of these networks is illustrated by using well-known model problems like the traveling salesman problem and the assignment problem. The ANNs are then subjected to up to 13 simultaneous stuck-at-1 or stuck-at-0 faults for network sizes of up to 900 neurons. The effect of these faults on the performance is demonstrated, and the cause for the observed fault-tolerance is discussed. An application is presented in which a network performs a critical task for a real-time distributed processing system by generating new task allocations during the reconfiguration of the system. The performance degradation of the ANN under the presence of faults is investigated by large-scale simulations and the potential benefits of delegating a critical task to a fault-tolerant network are discussed. >
---
paper_title: Enhanced MLP Performance and Fault Tolerance Resulting from Synaptic Weight Noise During Training

paper_content:

We analyze the effects of analog noise on the synaptic arithmetic during multilayer perceptron training, by expanding the cost function to include noise-mediated terms. Predictions are made in the light of these calculations that suggest that fault tolerance, training quality and training trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The results appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implications for all applications, particularly those involving "inaccurate" analog neural VLSI. >
---
paper_title: TOTEM: a digital processor for neural networks and Reactive Tabu Search

paper_content:

The Reactive Tabu Search (RTS) algorithm permits the training of neural networks with low number of bits per weight, low computational accuracy, no local minima “trapping”, and limited sensitivity to the initial conditions. Two architectures for the implementation of Multi-Layer Perceptrons based directly on the properties of RTS are presented. They are characterized by high-speed integer computation, tunable precision, 1/0 balanced design, internal storage of the weights and compact processing units. TOTEM, a silicon implementation which uses only 8 bits per weight and 16 bits per input, is then presented. A triggering problem in High Energy Physics is used as a test-bed for the algorithm and the processor. The eqected performance is briefly compared with those of two alternative processors.
---

Title: Hardware-friendly learning algorithms for neural networks: an overview

Section 1: Introduction
Description 1: Introduce the importance of hardware implementations in neural networks and briefly illustrate the categories of hardware-friendly learning algorithms.

Section 2: Hardware implementations of neural networks: problems and constraints
Description 2: Discuss the challenges and constraints associated with hardware implementations of neural networks, such as accuracy, area, and reliability.

Section 3: Hardware-friendly learning algorithms
Description 3: Present different hardware-friendly learning algorithms tailored for efficient implementation.

Section 3.1: Multilayer feedforward networks
Description 3.1: Review various approaches to making multilayer feedforward networks more hardware-friendly, including perturbation algorithms, local learning algorithms, and training without derivatives.

Section 3.2: Kohonen's self-organizing maps
Description 3.2: Examine adaptations to Kohonen's algorithm to facilitate its hardware implementation.

Section 3.3: Recurrent networks
Description 3.3: Discuss training methods for recurrent networks that are suitable for hardware implementation, such as the Boltzmann machine and mean field theory learning.

Section 3.4: Other types of neural networks
Description 3.4: Highlight other neural network models like RAM-based networks and cellular neural networks that are naturally suited for hardware implementation.

Section 4: Inaccuracy and robustness
Description 4: Address the issue of hardware inaccuracies and explore the robustness of neural network models under such conditions.

Section 4.1: Limited precision
Description 4.1: Investigate the effects of limited numerical precision and component imprecision on neural network performance.

Section 4.2: Robustness
Description 4.2: Discuss the resilience of neural networks to various types of faults and noise, and their impact on network performance.

Section 5: Summary and conclusions
Description 5: Summarize the various hardware-friendly algorithms and methods discussed, and reflect on their feasibility and robustness in actual hardware implementations.