Gradient calculation for dynamic recurrent neural networks: a survey
---
paper_title: The appeal of parallel distributed processing

paper_content:

This chapter contains sections titled: Parallel Distributed Processing, Examples Of PDP Models, Representation and Learning in PDP Models, Origins of Parallel Distributed Processing, Acknowledgments
---
paper_title: Cooperative computation of stereo disparity

paper_content:

Perhaps one of the most striking differences between a brain and today’s computers is the amount of “wiring.” In a digital computer the ratio of connections to components is about 3, whereas for the mammalian cortex it lies between 10 and 10,000 (1).
---
paper_title: Backpropagation Applied to Handwritten Zip Code Recognition

paper_content:

The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.
---
paper_title: Analysis of a cooperative stereo algorithm

paper_content:

Marr and Poggio (1976) recently described a cooperative algorithm that solves the correspondence problem for stereopsis. This article uses a probabilistic technique to analyze the convergence of that algorithm, and derives the conditions governing the stability of the solution state. The actual results of applying the algorithm to random-dot stereograms are compared with the probabilistic analysis. A satisfactory mathematical analysis of the asymptotic behaviour of the algorithm is possible for a suitable choice of the parameter values and loading rules, and again the actual performance of the algorithm under these conditions is compared with the theoretical predictions. Finally, some problems raised by the analysis of this type of "cooperative" algorithm are briefly discussed.
---
paper_title: Cooperative computation of stereo disparity

paper_content:

Perhaps one of the most striking differences between a brain and today’s computers is the amount of “wiring.” In a digital computer the ratio of connections to components is about 3, whereas for the mammalian cortex it lies between 10 and 10,000 (1).
---
paper_title: Analysis of a cooperative stereo algorithm

paper_content:

Marr and Poggio (1976) recently described a cooperative algorithm that solves the correspondence problem for stereopsis. This article uses a probabilistic technique to analyze the convergence of that algorithm, and derives the conditions governing the stability of the solution state. The actual results of applying the algorithm to random-dot stereograms are compared with the probabilistic analysis. A satisfactory mathematical analysis of the asymptotic behaviour of the algorithm is possible for a suitable choice of the parameter values and loading rules, and again the actual performance of the algorithm under these conditions is compared with the theoretical predictions. Finally, some problems raised by the analysis of this type of "cooperative" algorithm are briefly discussed.
---
paper_title: The appeal of parallel distributed processing

paper_content:

This chapter contains sections titled: Parallel Distributed Processing, Examples Of PDP Models, Representation and Learning in PDP Models, Origins of Parallel Distributed Processing, Acknowledgments
---
paper_title: Shape Recognition and Illusory Conjunctions

paper_content:

One way to achieve viewpoint-invariant shape recognition is to impose a canonical, object-based frame of reference on a shape and to describe the positions, sizes and orientations of the shape's features relative to the imposed frame. This compulation can be implemented in a parallel network of neuron-like processors, but the network has a tendency to make errors of a peculiar kind: When presented with several shapes it sometimes perceives one shape in the position of another. The parameters can be carefully tuned to avoid these "illusory conjunctions" in normal circumstances, but they reappear if the visual input is replaced by a random mask before the network has settled down. Treisman and Schmidt (1982) have shown that people make similar errors.
---
paper_title: FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling

paper_content:

A new neural network architecture involving either local feedforward global feedforward, and/or local recurrent global feedforward structure is proposed. A learning rule minimizing a mean square error criterion is derived. The performance of this algorithm (local recurrent global feedforward architecture) is compared with a local-feedforward global-feedforward architecture. It is shown that the local-recurrent global-feedforward model performs better than the local-feedforward global-feedforward model.
---
paper_title: Green's Function Method for Fast On-Line Learning Algorithm of Recurrent Neural Networks

paper_content:

The two well known learning algorithms of recurrent neural networks are the back-propagation (Rumelhart & et. al., Werbos) and the forward propagation (Williams and Zipser). The main drawback of back-propagation is its off-line backward path in time for error cumulation. This violates the on-line requirement in many practical applications. Although the forward propagation algorithm can be used in an on-line manner, the annoying drawback is the heavy computation load required to update the high dimensional sensitivity matrix (O(N4) operations for each time step). Therefore, to develop a fast forward algorithm is a challenging task. In this paper we proposed a forward learning algorithm which is one order faster (only O(N3) operations for each time step) than the sensitivity matrix algorithm. The basic idea is that instead of integrating the high dimensional sensitivity dynamic equation we solve forward in time for its Green's function to avoid the redundant computations, and then update the weights whenever the error is to be corrected. ::: ::: A Numerical example for classifying state trajectories using a recurrent network is presented. It substantiated the faster speed of the proposed algorithm than the Williams and Zipser's algorithm.
---
paper_title: Learning Complex, Extended Sequences Using the Principle of History Compression

paper_content:

Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to “divide and conquer” by recursively decomposing sequences. I describe two architectures. The first functions as a self-organizing multilevel hierarchy of recurrent networks. The second, involving only two recurrent networks, tries to collapse a multilevel predictor hierarchy into a single recurrent net. Experiments show that the system can require less computation per time step and many fewer training sequences than conventional training algorithms for recurrent nets.
---
paper_title: The appeal of parallel distributed processing

paper_content:

This chapter contains sections titled: Parallel Distributed Processing, Examples Of PDP Models, Representation and Learning in PDP Models, Origins of Parallel Distributed Processing, Acknowledgments
---

Title: Gradient Calculation for Dynamic Recurrent Neural Networks: A Survey

Section 1: Why Recurrent Networks
Description 1: Discuss the motivation for exploring recurrent neural networks and their capabilities with temporal behaviors.

Section 2: Continuous vs. Discrete Time
Description 2: Compare and contrast continuous time networks with discrete time systems, highlighting their respective advantages and use cases.

Section 3: Learning in Networks with Fixedpoints
Description 3: Outline learning algorithms for networks that converge to stable fixedpoints and discuss their computational relevance for constraint satisfaction and associative memory tasks.

Section 4: Will a Fixedpoint Exist?
Description 4: Discuss special cases and conditions that ensure recurrent networks converge to fixedpoints and explore the empirical and theoretical background supporting it.

Section 5: Problems with Fixedpoints
Description 5: Identify and explain potential issues with using fixedpoint learning algorithms, including non-continuous mapping and stability concerns.

Section 6: Computing the Gradient Without Assuming a Fixedpoint
Description 6: Explore gradient computation techniques for dynamic behaviors in recurrent neural networks, particularly focusing on Backpropagation Through Time (BPTT).

Section 7: Real Time Recurrent Learning
Description 7: Introduce the Real Time Recurrent Learning (RTRL) algorithm, detailing its computational approach and challenges.

Section 8: Less Computationally Burdensome Online Techniques
Description 8: Discuss alternative online learning techniques that reduce computational burden, including modified versions of BPTT and RTRL.

Section 9: Simulation of an Associative Network
Description 9: Provide an example simulation demonstrating recurrent backpropagation networks learning an associative task.

Section 10: Stability and Perturbation Experiments
Description 10: Analyze the stability of neural network behaviors, specifically investigating the effect of perturbations on network performance.

Section 11: Other Non-fixedpoint Techniques
Description 11: Review additional techniques for training recurrent networks without relying on fixedpoints, such as Elman networks, moving targets method, and others.

Section 12: Learning with Scale Parameters
Description 12: Explain the adaptation of gradient descent to handle scale parameters like time constants and time delays, along with their unique challenges and solutions.

Section 13: Complexity Comparison
Description 13: Compare the computational complexities of different recurrent network learning algorithms, both for sequential and online methods.

Section 14: Speeding the Optimization
Description 14: Discuss techniques for optimizing learning speed in recurrent networks, particularly focusing on handling the stiffness of the Hessian matrix.

Section 15: Conclusions
Description 15: Summarize the survey findings, emphasizing the practicality and variety of learning algorithms available for recurrent networks.