Cooperative recurrent modular neural networks for constrained optimization: a survey of models and applications
---
paper_title: A Modifled Neural Network for Quadratic Programming with Real-Time Applications

paper_content:

This paper presents a modifled neural network for solving strictly quadratic programming problems with general linear constraints. It is shown that the proposed neural network is globally convergent to a unique optimal solution within a flnite time. Compared with the existing primal-dual neural network and the dual neural network for solving such problems, the proposed neural network has a low complexity for implementation and can be guaranteed to have an exponential convergence rate. The good performance of the proposed neural network is illustrated by a real-time application to Kinematic Control.
---
paper_title: A neural network for robust LCMP beamforming

paper_content:

Calculating an optimal beamforming weight is a main task of beamforming. Quadratic constraints on the weight vector of an adaptive linearly constrained minimum power (LCMP) beamformer can improve robustness to pointing errors and to random perturbations in sensor parameters. This paper presents a neural network approach to the robust LCMP beamformer with the quadratic constraint. Compared with the existing neural networks for the LCMP beamformer, the proposed neural network converges fast to an optimal weight. Compared with the existing adaptive algorithms for the robust LCMP beamformer, in addition to parallel implementation, the proposed neural network is guaranteed to converge exponentially to an optimal weight. Simulations demonstrate that the proposed neural network has better interference suppression and faster convergence than the existing neural networks and the adaptive algorithms.
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: Neural networks for nonlinear programming

paper_content:

The dynamics of the modified canonical nonlinear programming circuit are studied and how to guarantee the stability of the network's solution. By considering the total cocontent function, the solution of the canonical nonlinear programming circuit is reconciled with the problem being modeled. In addition, it is shown how the circuit can be realized using a neural network, thereby extending the results of D.W. Tank and J.J. Hopefield (ibid., vol.CAS-33, p.533-41, May 1986) to the general nonlinear programming problem. >
---
paper_title: Lagrange programming neural networks

paper_content:

A class of neural networks appropriate for general nonlinear programming, i.e., problems including both equality and inequality constraints, is analyzed in detail. The methodology is based on the Lagrange multiplier theory in optimization and seeks to provide solutions satisfying the necessary conditions of optimality. The equilibrium point of the network satisfies the Kuhn-Tucker condition for the problem. No explicit restriction is imposed on the form of the cost function apart from some general regularity and convexity conditions. The stability of the neural networks is analyzed in detail. The transient behavior of the network is simulated and the validity of the approach is verified for a practical problem, maximum entropy image restoration. >
---
paper_title: A high performance neural network for solving nonlinear programming problems with hybrid constraints

paper_content:

Abstract A continuous neural network is proposed in this Letter for solving optimization problems. It not only can solve nonlinear programming problems with the constraints of equality and inequality, but also has a higher performance. The main advantage of the network is that it is an extension of Newton's gradient method for constrained problems, the dynamic behavior of the network under special constraints and the convergence rate can be investigated. Furthermore, the proposed network is simpler than the existing networks even for solving positive definite quadratic programming problems. The network considered is constrained by a projection operator on a convex set. The advanced performance of the proposed network is demonstrated by means of simulation of several numerical examples.
---
paper_title: Modified Projection-Type Methods For Monotone Variational Inequalities

paper_content:

We propose new methods for solving the variational inequality problem where the underlying function $F$ is monotone. These methods may be viewed as projection-type methods in which the projection direction is modified by a strongly monotone mapping of the form $I - \alpha F$ or, if $F$ is affine with underlying matrix $M$, of the form $I+ \alpha M^T$, with $\alpha \in (0,\infty)$. We show that these methods are globally convergent, and if in addition a certain error bound based on the natural residual holds locally, the convergence is linear. Computational experience with the new methods is also reported.
---
paper_title: A new neural network for solving linear programming problems and its application

paper_content:

Presents a new neural network which improves existing neural networks for solving general linear programming problems. The network, without setting parameter, uses only simple hardware in which no analog multipliers are required, and is proved to be completely stable to the exact solutions. Moreover, using this network the author can solve linear programming problems and its dual simultaneously, and cope with problems with nonunique solutions whose set is allowed to be unbounded.
---
paper_title: Modularity in neural computing

paper_content:

This paper considers neural computing models for information processing in terms of collections of subnetwork modules. Two approaches to generating such networks are studied. The first approach includes networks with functionally independent subnetworks, where each subnetwork is designed to have specific functions, communication, and adaptation characteristics. The second approach is based on algorithms that can actually generate network and subnetwork topologies, connections, and weights to satisfy specific constraints. Associated algorithms to attain these goals include evolutionary computation and self-organizing maps. We argue that this modular approach to neural computing is more in line with the neurophysiology of the vertebrate cerebral cortex, particularly with respect to sensation and perception. We also argue that this approach has the potential to aid in solutions to large-scale network computational problems - an identified weakness of simply defined artificial neural networks.
---
paper_title: Neural networks and physical systems with emergent collective computational abilities

paper_content:

Abstract ::: Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.
---
paper_title: Efficient classification for multiclass problems using modular neural networks

paper_content:

The rate of convergence of net output error is very low when training feedforward neural networks for multiclass problems using the backpropagation algorithm. While backpropagation will reduce the Euclidean distance between the actual and desired output vectors, the differences between some of the components of these vectors increase in the first iteration. Furthermore, the magnitudes of subsequent weight changes in each iteration are very small, so that many iterations are required to compensate for the increased error in some components in the initial iterations. Our approach is to use a modular network architecture, reducing a K-class problem to a set of K two-class problems, with a separately trained network for each of the simpler problems. Speedups of one order of magnitude have been obtained experimentally, and in some cases convergence was possible using the modular approach but not using a nonmodular network. >
---
paper_title: Simple “Neural” optimization networks: an A/D converter, signal decision circuit, and a linear programming circuit

paper_content:

We describe how several optimization problems can be rapidly solved by highly interconnected networks of simple analog processors. Analog-to-digital (A/D) conversion was considered as a simple optimization problem, and an A/D converter of novel architecture was designed. A/D conversion is a simple example of a more general class of signal-decision problems which we show could also be solved by appropriately constructed networks. Circuits to solve these problems were designed using general principles which result from an understanding of the basic collective computational properties of a specific class of analog-processor networks. We also show that a network which solves linear programming problems can be understood from the same concepts.
---
paper_title: Neural data fusion algorithms based on a linearly constrained least square method

paper_content:

Two novel neural data fusion algorithms based on a linearly constrained least square (LCLS) method are proposed. While the LCLS method is used to minimize the energy of the linearly fused information, two neural-network algorithms are developed to overcome the ill-conditioned and singular problems of the sample covariance matrix arisen in the LCLS method. The proposed neural fusion algorithms are samples for implementation using both software and hardware. Compared with the existing fusion methods, the proposed neural data fusion method has an unbiased statistical property and does not require any a priori knowledge about the noise covariance. It is shown that the proposed neural fusion algorithms converge globally to the optimal fusion solution when the sample covariance matrix is singular, and converge globally with exponential rate when the sample covariance matrix is nonsingular. We apply the proposed neural fusion method to image and signal fusion, and it is shown that the quality of the solution can be greatly enhanced by the proposed technique.
---
paper_title: Minimum fuel neural networks and their applications to overcomplete signal representations

paper_content:

The overcomplete signal representation (OSR) is a recently established adaptive signal representation method. As an adaptive signal representation method, the OSR means that a given signal is decomposed onto a number of optimal basis components, which are found from an overcomplete basis dictionary via some optimization algorithms, such as the matching pursuit (MP), method of frame (MOF) and basis pursuit (BP). Such ideas are actually very close to or exactly the same as solving a minimum fuel (MF) problem. The MF problem is a well-established minimum L/sub 1/-norm optimization model with linear constraints. The BP-based OSR proposed by Chen and Donoho is exactly the same model as the MF model. The work of Chen and Donoho showed that the MF model could be used as a generalized method for solving an OSR problem and it outperformed the MP and the MOF. In this paper, the neural implementation of the MF model and its applications to the OSR are presented. A new neural network, namely the minimum fuel neural network (MFNN), is constructed and its convergence in solving the MP problem is proven theoretically and validated experimentally. Compared with the implementation of the original BP, the MFNN does not double the scales of the problem and its convergence is independent of initial conditions. It is shown that the MFNN is promising for the application in the OSR's of various kinds of nonstationary signals with a high time-frequency resolution and feasibility of real-time implementation. As an extension, a two-dimensional (2-D) MF model suitable for image data compression is also proposed and its neural implementation is presented.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: A modular neural network for control of mobile robots

paper_content:

A new modular neural network architecture and its learning algorithm are proposed for a mobile robot controller. The learning algorithm for the proposed new network architecture is based on a feedback error learning procedure, which requires a feedback controller for training processes. It is not so easy, however, to obtain a robot feedback controller, when the robot control task is much more complex. In the present architecture, the complex robot control task is divided into a couple of small simple tasks, each of which is assigned to each of small network modules, respectively. By dividing the complex task, the simple feedback controllers are assigned to the network modules. Therefore, the neural network in each module can be trained by the feedback error learning scheme. The command to the robots is the weighted sum of the outputs of the modules. The weights for each module are obtained from a neural network which is one of the network modules in our proposed architecture. The present neural network architecture and learning algorithm are applied to a set of several robot controllers, whose task is to push a large box to a goal. It is confirmed through computer simulation experiments that the algorithm can train the robot controller skillfully.
---
paper_title: The Neural Network Model for Solving Minimax Problems with Constraints

paper_content:

This paper presents a kind of neural network models for solving quadratic minimax problems with constraints, and shows its global asymptotic stability by using LaSalle invariant principle. The L 1 norm minimizing problems are considered as applications.
---
paper_title: Neural network ensembles: combining multiple models for enhanced performance using a multistage approach

paper_content:

: Neural network ensembles (sometimes referred to as committees or classifier ensembles) are effective techniques to improve the generalization of a neural network system. Combining a set of neural network classifiers whose error distributions are diverse can generate better results than any single classifier. In this paper, some methods for creating ensembles are reviewed, including the following approaches: methods of selecting diverse training data from the original source data set, constructing different neural network models, selecting ensemble nets from ensemble candidates and combining ensemble members' results. In addition, new results on ensemble combination methods are reported.
---
paper_title: CMNN: cooperative modular neural network

paper_content:

The current generation of nonmodular neural network classifiers is unable to cope with classification problems which have a wide range of overlap among classes. This is due to the high coupling among the networks' hidden nodes. We propose the Cooperative Modular Neural Network (CMNN) architecture, which deals with different levels of overlap in different modules. The modules share their information and cooperate in taking a global classification decision through voting. Moreover, special modules are dedicated to resolve high overlaps in the input-space. The performance of the new model outperforms that of the nonmodular alternative when when applied to ten famous benchmark classification problems.
---
paper_title: A new gradient-based neural network for solving linear and quadratic programming problems

paper_content:

A new gradient-based neural network is constructed on the basis of the duality theory, optimization theory, convex analysis theory, Lyapunov stability theory, and LaSalle invariance principle to solve linear and quadratic programming problems. In particular, a new function F(x, y) is introduced into the energy function E(x, y) such that the function E(x, y) is convex and differentiable, and the resulting network is more efficient. This network involves all the relevant necessary and sufficient optimality conditions for convex quadratic programming problems. For linear programming and quadratic programming (QP) problems with unique and infinite number of solutions, we have proven strictly that for any initial point, every trajectory of the neural network converges to an optimal solution of the QP and its dual problem. The proposed network is different from the existing networks which use the penalty method or Lagrange method, and the inequality constraints are properly handled. The simulation results show that the proposed neural network is feasible and efficient.
---
paper_title: A Sequential Quadratically Constrained Quadratic Programming Method for Differentiable Convex Minimization

paper_content:

This paper presents a sequential quadratically constrained quadratic programming (SQCQP) method for solving smooth convex programs. The SQCQP method solves at each iteration a subproblem that involves convex quadratic inequality constraints as well as a convex quadratic objective function. Such a quadratically constrained quadratic programming problem can be formulated as a second-order cone program, which can be solved efficiently by using interior point methods. We consider the following three fundamental issues on the SQCQP method: the feasibility of subproblems, the global convergence, and the quadratic rate of convergence. In particular, we show that the Maratos effect is avoided without any modification to the search direction, even though we use an ordinary $\ell_1$ exact penalty function as the line search merit function.
---
paper_title: Sliding modes in solving convex programming problems

paper_content:

Sliding modes are used to analyze a class of dynamical systems that solve convex programming problems. The analysis is carried out using concepts from the theory of differential equations with discontinuous right-hand sides and Lyapunov stability theory. It is shown that the equilibrium points of the system coincide with the minimizers of the convex programming problem, and that irrespective of the initial state of the system the state trajectory converges to the solution set of the problem. The dynamic behavior of the systems is illustrated by two numerical examples.
---
paper_title: Nonlinear switched capacitor 'neural' networks for optimization problems

paper_content:

A systematic approach is presented for the design of analog neural nonlinear programming solvers using switched-capacitor (SC) integrated circuit techniques. The method is based on formulating a dynamic gradient system whose state evolves in time toward the solution point of the corresponding programming problem. A neuron cell for the linear and the quadratic problem suitable for monolithic implementation is introduced. The design of this neuron and its corresponding synapses using SC techniques is considered in detail. An SC circuit architecture based on a reduced set of basic building blocks with high modularity is presented. Simulation results using a mixed-mode simulator (DIANA) and experimental results from breadboard prototypes are included, illustrating the validity of the proposed techniques. >
---
paper_title: An Extended Projection Neural Network for Constrained Optimization

paper_content:

Recently, a projection neural network has been shown to be a promising computational model for solving variational inequality problems with box constraints. This letter presents an extended projection neural network for solving monotone variational inequality problems with linear and nonlinear constraints. In particular, the proposed neural network can include the projection neural network as a special case. Compared with the modified projection-type methods for solving constrained monotone variational inequality problems, the proposed neural network has a lower complexity and is suitable for parallel implementation. Furthermore, the proposed neural network is theoretically proven to be exponentially convergent to an exact solution without a Lipschitz condition. Illustrative examples show that the extended projection neural network can be used to solve constrained monotone variational inequality problems.
---
paper_title: Learning decision fusion in cooperative modular neural networks

paper_content:

The modular neural network offers several advantages over classical non-modular neural network approaches to complex pattern classification problems. However, the accuracy of the modular approach depends greatly on the accurate fusion of the individual classification decisions. The paper presents a method for improving the overall accuracy of modular neural networks by incorporating an adaptive decision fusion mechanism. The proposed algorithm offers significant improvement over typical modular networks by evolving a more informed decision fusion mechanism that can greatly improve the final classification decision for complex classification tasks.
---
paper_title: Neural network for solving linear programming problems with bounded variables

paper_content:

A new neural network for solving linear programming problems with bounded variables is presented. The network is shown to be completely stable and globally convergent to the solutions to the linear programming problems. The proposed new network is capable of achieving the exact solutions, in contrast to existing optimization neural networks which need a suitable choice of the network parameters and thus can obtain only approximate solutions. Furthermore, both the primal problems and their dual problems are solved simultaneously by the new network. >
---
paper_title: A Dual Neural Network for Redundancy Resolution of Kinematically Redundant Manipulators Subject to Joint Limits and Joint Velocity Limits

paper_content:

In this paper, a recurrent neural network called the dual neural network is proposed for online redundancy resolution of kinematically redundant manipulators. Physical constraints such as joint limits and joint velocity limits, together with the drift-free criterion as a secondary task, are incorporated into the problem formulation of redundancy resolution. Compared to other recurrent neural networks, the dual neural network is piecewise linear and has much simpler architecture with only one layer of neurons. The dual neural network is shown to be globally (exponentially) convergent to optimal solutions. The dual neural network is simulated to control the PA10 robot manipulator with effectiveness demonstrated.
---
paper_title: A Constructive Algorithm for Training Cooperative Neural Network Ensembles

paper_content:

Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability.
---
paper_title: Recursive structure estimation for nonlinear identification with modular networks

paper_content:

The paper presents a recursive nonlinear identification scheme with modular networks consisting of local linear models. New local linear models are added online as and when necessary. The algorithms is developed within a probabilistic framework and utilises the Kalman filter for estimation of model parameters. Simulated results demonstrate the operation of the algorithm.
---
paper_title: A projection neural network and its application to constrained optimization problems

paper_content:

In this paper, we present a recurrent neural network for solving the nonlinear projection formulation. It is shown here that the proposed neural network is stable in the sense of Lyapunov and globally convergent, globally asymptotically stable, and globally exponentially stable, respectively under different conditions. Compared with the existing neural network for solving the projection formulation, the proposed neural network has a single-layer structure and is amenable to parallel implementation. Moreover, the proposed neural network has no Lipschitz condition, and, thus can be applied to solve a very broad class of constrained optimization problems that are special cases of the nonlinear projection formulation. Simulation shows that the proposed neural network is effective in solving these constrained optimization problems.
---
paper_title: Global convergence analysis of Lagrangian networks

paper_content:

Many models arisen in signal and image processing can be formulated as a nonlinear convex programming problem with linear equality constraints. A Lagrangian network was developed for real-time applications of these problems. Yet, the global convergence of the Lagrangian network has not been well studied due to the asymmetry of the corresponding Lagrange system. In this brief, based on a new Lyapunov function we analyze and prove the global convergence of the Lagrangian network. Simulation examples are provided to show the effectiveness of the obtained results.
---
paper_title: Modular network control for robot manipulator

paper_content:

Plant dynamics of highly nonlinear dynamical systems often vary over its parameter space. This generates a natural partitioning of the parameter space based upon the operating points of the plant. As a result, it may be wise to generate local control strategies at, the operating points rather than a single global strategy. This article describes a modular neural network architecture that generates a piece-wise continuous control strategy designed to fuse local strategies together to form a single strategy. The modularity is achieved through a gating network that controls the competition and cooperation of local experts. The gating network is a high order dynamical system network, while the local expert is a multi-layer feedforward network. The capability of this technique is demonstrated by building a neurocontroller for the two-link robot manipulator with two revolute joints.
---
paper_title: Multi-modular neural network-hybrid architectures: a review

paper_content:

A formalism to train multi-modular architectures (MMAs) is given. Examples of MMAs for various applications in image and speech processing are shown, which illustrate the benefits of using MMAs.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Neural network ensembles: combining multiple models for enhanced performance using a multistage approach

paper_content:

: Neural network ensembles (sometimes referred to as committees or classifier ensembles) are effective techniques to improve the generalization of a neural network system. Combining a set of neural network classifiers whose error distributions are diverse can generate better results than any single classifier. In this paper, some methods for creating ensembles are reviewed, including the following approaches: methods of selecting diverse training data from the original source data set, constructing different neural network models, selecting ensemble nets from ensemble candidates and combining ensemble members' results. In addition, new results on ensemble combination methods are reported.
---
paper_title: Modularity in neural computing

paper_content:

This paper considers neural computing models for information processing in terms of collections of subnetwork modules. Two approaches to generating such networks are studied. The first approach includes networks with functionally independent subnetworks, where each subnetwork is designed to have specific functions, communication, and adaptation characteristics. The second approach is based on algorithms that can actually generate network and subnetwork topologies, connections, and weights to satisfy specific constraints. Associated algorithms to attain these goals include evolutionary computation and self-organizing maps. We argue that this modular approach to neural computing is more in line with the neurophysiology of the vertebrate cerebral cortex, particularly with respect to sensation and perception. We also argue that this approach has the potential to aid in solutions to large-scale network computational problems - an identified weakness of simply defined artificial neural networks.
---
paper_title: Modular Neural Network Classifiers: A Comparative Study

paper_content:

There is a wide variety of Modular Neural Network (MNN) classifiers in the literature. They differ according to the design of their architecture, task-decomposition scheme, learning procedure, and multi-module decision-making strategy. Meanwhile, there is a lack of comparative studies in the MNN literature. This paper compares ten MNN classifiers which give a good representation of design varieties, viz., Decoupled; Other-output; ART-BP; Hierarchical; Multiple-experts; Ensemble (majority vote); Ensemble (average vote); Merge-glue; Hierarchical Competitive Neural Net; and Cooperative Modular Neural Net. Two benchmark applications of different degree and nature of complexity are used for performance comparison, and the strength-points and drawbacks of the different networks are outlined. The aim is to help a potential user to choose an appropriate model according to the application in hand and the available computational resources.
---
paper_title: A Constructive Algorithm for Training Cooperative Neural Network Ensembles

paper_content:

Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability.
---
paper_title: Neural networks for nonlinear programming

paper_content:

The dynamics of the modified canonical nonlinear programming circuit are studied and how to guarantee the stability of the network's solution. By considering the total cocontent function, the solution of the canonical nonlinear programming circuit is reconciled with the problem being modeled. In addition, it is shown how the circuit can be realized using a neural network, thereby extending the results of D.W. Tank and J.J. Hopefield (ibid., vol.CAS-33, p.533-41, May 1986) to the general nonlinear programming problem. >
---
paper_title: Nonlinear Programming -- Theory and Algorithms

paper_content:

Nonlinear Programming presents recent developments of key topics in nonlinear programming (NLP) using a logical and self-contained format. The volume is divided into three sections: convex analysis, optimality conditions, and dual computational techniques. Precise statements of algorithms are given along with convergence analysis. Each chapter contains detailed numerical examples, graphical illustrations, and numerous exercises to aid readers in understanding the concepts and methods discussed.
---
paper_title: Lagrange programming neural networks

paper_content:

A class of neural networks appropriate for general nonlinear programming, i.e., problems including both equality and inequality constraints, is analyzed in detail. The methodology is based on the Lagrange multiplier theory in optimization and seeks to provide solutions satisfying the necessary conditions of optimality. The equilibrium point of the network satisfies the Kuhn-Tucker condition for the problem. No explicit restriction is imposed on the form of the cost function apart from some general regularity and convexity conditions. The stability of the neural networks is analyzed in detail. The transient behavior of the network is simulated and the validity of the approach is verified for a practical problem, maximum entropy image restoration. >
---
paper_title: A novel neural network for variational inequalities with linear and nonlinear constraints

paper_content:

Variational inequality is a uniform approach for many important optimization and equilibrium problems. Based on the sufficient and necessary conditions of the solution, this paper presents a novel neural network model for solving variational inequalities with linear and nonlinear constraints. Three sufficient conditions are provided to ensure that the proposed network with an asymmetric mapping is stable in the sense of Lyapunov and converges to an exact solution of the original problem. Meanwhile, the proposed network with a gradient mapping is also proved to be stable in the sense of Lyapunov and to have a finite-time convergence under some mild conditions by using a new energy function. Compared with the existing neural networks, the new model can be applied to solve some nonmonotone problems, has no adjustable parameter, and has lower complexity. Thus, the structure of the proposed network is very simple. Since the proposed network can be used to solve a broad class of optimization problems, it has great application potential. The validity and transient behavior of the proposed neural network are demonstrated by several numerical examples.
---
paper_title: Review of pseudoinverse control for use with kinematically redundant manipulators

paper_content:

Kinematically redundant manipulators have a number of potential advantages over current manipulator designs. For this type of arm, velocity control through pseudoinverses is suggested. Questions associated with pseudoinverse control are examined in detail and show that in some cases this control leads to undesired arm configurations. A method for distributing joint angles of a redundant arm in a good approximation to a true minimax criterion is described. In addition several numerical considerations are discussed.
---
paper_title: A high performance neural network for solving nonlinear programming problems with hybrid constraints

paper_content:

Abstract A continuous neural network is proposed in this Letter for solving optimization problems. It not only can solve nonlinear programming problems with the constraints of equality and inequality, but also has a higher performance. The main advantage of the network is that it is an extension of Newton's gradient method for constrained problems, the dynamic behavior of the network under special constraints and the convergence rate can be investigated. Furthermore, the proposed network is simpler than the existing networks even for solving positive definite quadratic programming problems. The network considered is constrained by a projection operator on a convex set. The advanced performance of the proposed network is demonstrated by means of simulation of several numerical examples.
---
paper_title: A general methodology for designing globally convergent optimization neural networks

paper_content:

We present a general methodology for designing optimization neural networks. We prove that the neural networks constructed by using the proposed method are guaranteed to be globally convergent to solutions of problems with bounded or unbounded solution sets, in contrast with the gradient methods whose convergence is not guaranteed. We show that the proposed method contains both the gradient methods and nongradient methods employed in existing optimization neural networks as special cases. Based on the theoretical results of the proposed method, we study the convergence and stability of general gradient models in the case of unisolated solutions. Using the proposed method, we derive some new neural network models for a very large class of optimization problems, in which the equilibrium points correspond to exact solutions and there is no variable parameter. Finally, some numerical examples show the effectiveness of the method.
---
paper_title: A new neural network for solving linear programming problems and its application

paper_content:

Presents a new neural network which improves existing neural networks for solving general linear programming problems. The network, without setting parameter, uses only simple hardware in which no analog multipliers are required, and is proved to be completely stable to the exact solutions. Moreover, using this network the author can solve linear programming problems and its dual simultaneously, and cope with problems with nonunique solutions whose set is allowed to be unbounded.
---
paper_title: A new gradient-based neural network for solving linear and quadratic programming problems

paper_content:

A new gradient-based neural network is constructed on the basis of the duality theory, optimization theory, convex analysis theory, Lyapunov stability theory, and LaSalle invariance principle to solve linear and quadratic programming problems. In particular, a new function F(x, y) is introduced into the energy function E(x, y) such that the function E(x, y) is convex and differentiable, and the resulting network is more efficient. This network involves all the relevant necessary and sufficient optimality conditions for convex quadratic programming problems. For linear programming and quadratic programming (QP) problems with unique and infinite number of solutions, we have proven strictly that for any initial point, every trajectory of the neural network converges to an optimal solution of the QP and its dual problem. The proposed network is different from the existing networks which use the penalty method or Lagrange method, and the inequality constraints are properly handled. The simulation results show that the proposed neural network is feasible and efficient.
---
paper_title: Sliding modes in solving convex programming problems

paper_content:

Sliding modes are used to analyze a class of dynamical systems that solve convex programming problems. The analysis is carried out using concepts from the theory of differential equations with discontinuous right-hand sides and Lyapunov stability theory. It is shown that the equilibrium points of the system coincide with the minimizers of the convex programming problem, and that irrespective of the initial state of the system the state trajectory converges to the solution set of the problem. The dynamic behavior of the systems is illustrated by two numerical examples.
---
paper_title: Nonlinear switched capacitor 'neural' networks for optimization problems

paper_content:

A systematic approach is presented for the design of analog neural nonlinear programming solvers using switched-capacitor (SC) integrated circuit techniques. The method is based on formulating a dynamic gradient system whose state evolves in time toward the solution point of the corresponding programming problem. A neuron cell for the linear and the quadratic problem suitable for monolithic implementation is introduced. The design of this neuron and its corresponding synapses using SC techniques is considered in detail. An SC circuit architecture based on a reduced set of basic building blocks with high modularity is presented. Simulation results using a mixed-mode simulator (DIANA) and experimental results from breadboard prototypes are included, illustrating the validity of the proposed techniques. >
---
paper_title: Neural network for solving linear programming problems with bounded variables

paper_content:

A new neural network for solving linear programming problems with bounded variables is presented. The network is shown to be completely stable and globally convergent to the solutions to the linear programming problems. The proposed new network is capable of achieving the exact solutions, in contrast to existing optimization neural networks which need a suitable choice of the network parameters and thus can obtain only approximate solutions. Furthermore, both the primal problems and their dual problems are solved simultaneously by the new network. >
---
paper_title: A new neural network for solving linear and quadratic programming problems

paper_content:

A new neural network for solving linear and quadratic programming problems is presented and is shown to be globally convergent. The new neural network improves existing neural networks for solving these problems: it avoids the parameter turning problem, it is capable of achieving the exact solutions, and it uses only simple hardware in which no analog multipliers for variables are required. Furthermore, the network solves both the primal problems and their dual problems simultaneously.
---
paper_title: Global convergence analysis of Lagrangian networks

paper_content:

Many models arisen in signal and image processing can be formulated as a nonlinear convex programming problem with linear equality constraints. A Lagrangian network was developed for real-time applications of these problems. Yet, the global convergence of the Lagrangian network has not been well studied due to the asymmetry of the corresponding Lagrange system. In this brief, based on a new Lyapunov function we analyze and prove the global convergence of the Lagrangian network. Simulation examples are provided to show the effectiveness of the obtained results.
---
paper_title: Novel Cooperative Neural Fusion Algorithms for Image Restoration and Image Fusion

paper_content:

To deal with the problem of restoring degraded images with non-Gaussian noise, this paper proposes a novel cooperative neural fusion regularization (CNFR) algorithm for image restoration. Compared with conventional regularization algorithms for image restoration, the proposed CNFR algorithm can relax need of the optimal regularization parameter to be estimated. Furthermore, to enhance the quality of restored images, this paper presents a cooperative neural fusion (CNF) algorithm for image fusion. Compared with existing signal-level image fusion algorithms, the proposed CNF algorithm can greatly reduce the loss of contrast information under blind Gaussian noise environments. The performance analysis shows that the proposed two neural fusion algorithms can converge globally to the robust and optimal image estimate. Simulation results confirm that in different noise environments, the proposed two neural fusion algorithms can obtain a better image estimate than several well known image restoration and image fusion methods
---
paper_title: A new neural network for solving linear programming problems and its application

paper_content:

Presents a new neural network which improves existing neural networks for solving general linear programming problems. The network, without setting parameter, uses only simple hardware in which no analog multipliers are required, and is proved to be completely stable to the exact solutions. Moreover, using this network the author can solve linear programming problems and its dual simultaneously, and cope with problems with nonunique solutions whose set is allowed to be unbounded.
---
paper_title: The Neural Network Model for Solving Minimax Problems with Constraints

paper_content:

This paper presents a kind of neural network models for solving quadratic minimax problems with constraints, and shows its global asymptotic stability by using LaSalle invariant principle. The L 1 norm minimizing problems are considered as applications.
---
paper_title: Support-vector networks

paper_content:

Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: Modular Neural Network Classifiers: A Comparative Study

paper_content:

There is a wide variety of Modular Neural Network (MNN) classifiers in the literature. They differ according to the design of their architecture, task-decomposition scheme, learning procedure, and multi-module decision-making strategy. Meanwhile, there is a lack of comparative studies in the MNN literature. This paper compares ten MNN classifiers which give a good representation of design varieties, viz., Decoupled; Other-output; ART-BP; Hierarchical; Multiple-experts; Ensemble (majority vote); Ensemble (average vote); Merge-glue; Hierarchical Competitive Neural Net; and Cooperative Modular Neural Net. Two benchmark applications of different degree and nature of complexity are used for performance comparison, and the strength-points and drawbacks of the different networks are outlined. The aim is to help a potential user to choose an appropriate model according to the application in hand and the available computational resources.
---
paper_title: Successive overrelaxation for support vector machines

paper_content:

Successive overrelaxation (SOR) for symmetric linear complementarity problems and quadratic programs is used to train a support vector machine (SVM) for discriminating between the elements of two massive datasets, each with millions of points. Because SOR handles one point at a time, similar to Platt's sequential minimal optimization (SMO) algorithm (1999) which handles two constraints at a time and Joachims' SVM/sup light/ (1998) which handles a small number of points at a time, SOR can process very large datasets that need not reside in memory. The algorithm converges linearly to a solution. Encouraging numerical results are presented on datasets with up to 10 000 000 points. Such massive discrimination problems cannot be processed by conventional linear or quadratic programming methods, and to our knowledge have not been solved by other methods. On smaller problems, SOR was faster than SVM/sup light/ and comparable or faster than SMO.
---
paper_title: Neural data fusion algorithms based on a linearly constrained least square method

paper_content:

Two novel neural data fusion algorithms based on a linearly constrained least square (LCLS) method are proposed. While the LCLS method is used to minimize the energy of the linearly fused information, two neural-network algorithms are developed to overcome the ill-conditioned and singular problems of the sample covariance matrix arisen in the LCLS method. The proposed neural fusion algorithms are samples for implementation using both software and hardware. Compared with the existing fusion methods, the proposed neural data fusion method has an unbiased statistical property and does not require any a priori knowledge about the noise covariance. It is shown that the proposed neural fusion algorithms converge globally to the optimal fusion solution when the sample covariance matrix is singular, and converge globally with exponential rate when the sample covariance matrix is nonsingular. We apply the proposed neural fusion method to image and signal fusion, and it is shown that the quality of the solution can be greatly enhanced by the proposed technique.
---
paper_title: A Generalized Least Absolute Deviation Method for Parameter Estimation of Autoregressive Signals

paper_content:

This paper proposes a generalized least absolute deviation (GLAD) method for parameter estimation of autoregressive (AR) signals under non-Gaussian noise environments. The proposed GLAD method can improve the accuracy of the estimation of the conventional least absolute deviation (LAD) method by minimizing a new cost function with parameter variables and noise error variables. Compared with second- and high-order statistical methods, the proposed GLAD method can obtain robustly an optimal AR parameter estimation without requiring the measurement noise to be Gaussian. Moreover, the proposed GLAD method can be implemented by a cooperative neural network (NN) which is shown to converge globally to the optimal AR parameter estimation within a finite time. Simulation results show that the proposed GLAD method can obtain more accurate estimates than several well-known estimation methods in the presence of different noise distributions.
---
paper_title: Neural data fusion algorithms based on a linearly constrained least square method

paper_content:

Two novel neural data fusion algorithms based on a linearly constrained least square (LCLS) method are proposed. While the LCLS method is used to minimize the energy of the linearly fused information, two neural-network algorithms are developed to overcome the ill-conditioned and singular problems of the sample covariance matrix arisen in the LCLS method. The proposed neural fusion algorithms are samples for implementation using both software and hardware. Compared with the existing fusion methods, the proposed neural data fusion method has an unbiased statistical property and does not require any a priori knowledge about the noise covariance. It is shown that the proposed neural fusion algorithms converge globally to the optimal fusion solution when the sample covariance matrix is singular, and converge globally with exponential rate when the sample covariance matrix is nonsingular. We apply the proposed neural fusion method to image and signal fusion, and it is shown that the quality of the solution can be greatly enhanced by the proposed technique.
---
paper_title: Improved neural network for SVM learning

paper_content:

The recurrent network of Xia et al. (1996) was proposed for solving quadratic programming problems and was recently adapted to support vector machine (SVM) learning by Tan et al. (2000). We show that this formulation contains some unnecessary circuits which, furthermore, can fail to provide the correct value of one of the SVM parameters and suggest how to avoid these drawbacks.
---
paper_title: A network of networks processing model for image regularization

paper_content:

We introduce a network of networks (NoN) model to solve image regularization problems. The method is motivated by the fact that natural image formation involves both local processing and globally coordinated parallel processing. Both forms are readily implemented using an NoN architecture. The modeling is very powerful in that it achieves high-quality adaptive processing, and it reduces the computational difference between inhomogeneous and homogeneous conditions. This method is able to provide fast, quality imaging in early vision, and its replicating structure and sparse connectivity readily lend themselves to hardware implementations.
---
paper_title: Novel Cooperative Neural Fusion Algorithms for Image Restoration and Image Fusion

paper_content:

To deal with the problem of restoring degraded images with non-Gaussian noise, this paper proposes a novel cooperative neural fusion regularization (CNFR) algorithm for image restoration. Compared with conventional regularization algorithms for image restoration, the proposed CNFR algorithm can relax need of the optimal regularization parameter to be estimated. Furthermore, to enhance the quality of restored images, this paper presents a cooperative neural fusion (CNF) algorithm for image fusion. Compared with existing signal-level image fusion algorithms, the proposed CNF algorithm can greatly reduce the loss of contrast information under blind Gaussian noise environments. The performance analysis shows that the proposed two neural fusion algorithms can converge globally to the robust and optimal image estimate. Simulation results confirm that in different noise environments, the proposed two neural fusion algorithms can obtain a better image estimate than several well known image restoration and image fusion methods
---
paper_title: A study of methods of choosing the smoothing parameter in image restoration by regularization

paper_content:

The method of regularization is portrayed as providing a compromise between fidelity to the data and smoothness, with the tradeoff being determined by a scalar smoothing parameter. Various ways of choosing this parameter are discussed in the case of quadratic regularization criteria. They are compared algebraically, and their statistical properties are comparatively assessed from the results of all extensive simulation study based on simple images. >
---
paper_title: Image restoration using a modified Hopfield network

paper_content:

A modified Hopfield neural network model for regularized image restoration is presented. The proposed network allows negative autoconnections for each neuron. A set of algorithms using the proposed neural network model is presented, with various updating modes: sequential updates; n-simultaneous updates; and partially asynchronous updates. The sequential algorithm is shown to converge to a local minimum of the energy function after a finite number of iterations. Since an algorithm which updates all n neurons simultaneously is not guaranteed to converge, a modified algorithm is presented, which is called a greedy algorithm. Although the greedy algorithm is not guaranteed to converge to a local minimum, the l/sub 1/ norm of the residual at a fixed point is bounded. A partially asynchronous algorithm is presented, which allows a neuron to have a bounded time delay to communicate with other neurons. Such an algorithm can eliminate the synchronization overhead of synchronous algorithms. >
---
paper_title: Iterative Regularized Least-Mean Mixed-Norm Image Restoration

paper_content:

We develop a regularized mixed-norm image restoration algorithm to deal with various types of noise. A mixed-norm functional is introduced, which combines the least mean square (LMS) and the least mean fourth (LMF) functionals, as well as a smoothing functional. Two regularization parameters are introduced: one to determine the relative importance of the LMS and LMF functionals, which is a function of the kurtosis, and another to determine the relative importance of the smoothing functional. The two parameters are chosen in such a way that the proposed functional is convex, so that a unique minimizer exists. An iterative algorithm is utilized for obtaining the solution, and its convergence is analyzed. The novelty of the proposed algorithm is that no knowledge of the noise distribution is required, and the relative contributions of the LMS, the LMF, and the smoothing functionals are adjusted based on the partially restored image. Experimental results demonstrate the effectiveness of the proposed algorithm.
---
paper_title: General iterative method of restoring linearly degraded images

paper_content:

A generalized iterative restoration algorithm for linearly degraded images is presented, based on the singular value decomposition of the degradation operator. Covergence of the algorithm is accelerated by imposing constraints on the solution. Realizations of the restoration algorithm for various types of degradation are presented, and the analogy to a modified version of the method by Gerchberg [ Opt. Acta21, 709 ( 1974)] and Papoulis [ IEEE Trans. Circuits Syst.CAS-22, 735 ( 1975)] is discussed. Results from computer-simulated images are given, and the effects of noise on the restored images are considered. The performance of the algorithm is found to be superior to that of pseudoinverse techniques.
---
paper_title: A class of robust entropic functionals for image restoration.

paper_content:

This paper considers the concept of robust estimation in regularized image restoration. Robust functionals are employed for the representation of both the noise and the signal statistics. Such functionals allow the efficient suppression of a wide variety of noise processes and permit the reconstruction of sharper edges than their quadratic counterparts. A new class of robust entropic functionals is introduced, which operates only on the high-frequency content of the signal and reflects sharp deviations in the signal distribution. This class of functionals can also incorporate prior structural information regarding the original image, in a way similar to the maximum information principle. The convergence properties of robust iterative algorithms are studied for continuously and noncontinuously differentiable functionals. The definition of the robust approach is completed by introducing a method for the optimal selection of the regularization parameter. This method utilizes the structure of robust estimators that lack analytic specification. The properties of robust algorithms are demonstrated through restoration examples in different noise environments.
---
paper_title: Hopfield neural network based algorithms for image restoration and reconstruction. I. Algorithms and simulations

paper_content:

In our previous work, the eliminating-highest error (EHE) criterion was proposed for the modified Hopfield (1982) neural network (MHNN) for image restoration and reconstruction. The performance of the MHNN is considerably improved by the EHE criterion as shown in many simulations. In inspiration of revealing the insight of the EHE criterion, in this paper, we first present a generalized updating rule (GUR) of the MHNN for gray image recovery. The stability properties of the GUR are given. It is shown that the neural threshold set up in this GUR is necessary and sufficient for energy decrease with probability one at each update. The new fastest-energy-descent (FED) criterion is then proposed parallel to the EHE criterion. While the EHE criterion is shown to achieve the highest probability of correct transition, the FED criterion achieves the largest amount of energy descent. In image restoration, the EHE and FED criteria are equivalent. A group of new algorithms based on the EHE and FED criteria is set up. A new measure, the correct transition rate (CTR), is proposed for the performance of iterative algorithms. Simulation results for gray image restoration show that the EHE (FED) based algorithms obtained the best visual quality and highest SNR of recovered images, took much smaller number of iterations, and had higher CTR. The CTR is shown to be a rational performance measure of iterative algorithms and predict quality of recovered images.
---
paper_title: A regularized iterative image restoration algorithm

paper_content:

The development of the algorithm is based on a set theoretic approach to regularization. Deterministic and/or statistical information about the undistorted image and statistical information about the noise are directly incorporated into the iterative procedure. The restored image is the center of an ellipsoid bounding the intersection of two ellipsoids. The proposed algorithm, which has the constrained least squares algorithm as a special case, is extended into an adaptive iterative restoration algorithm. The spatial adaptivity is introduced to incorporate properties of the human visual system. Convergence of the proposed iterative algorithms is established. For the experimental results which are shown, the adaptively restored images have better quality than the nonadaptively restored ones based on visual observations and on an objective criterion of merit which accounts for the noise masking property of the visual system. >
---
paper_title: A partitioned modified Hopfield neural network algorithm for real-time image restoration

paper_content:

Abstract In this paper, two image partitioning schemes are examined. The first scheme examined avoids boundary conflicts by the use of four restoration phases. The second scheme examined requires a degree of synchronization of the processors restoring adjacent regions. Both schemes avoid conflicting boundary conditions by taking into account the local image formation properties. Without any loss of processing speed, or increase in the number of processors required to restore the image, synchronizing conditions are not required in the four-phase scheme to restore the image accurately, however can be used to maximize restoration efficiency. An improved modified Hopfield neural network-based algorithm is developed to be especially applicable to the problems of real-time image processing based on the described partitioning schemes. The proposed algorithm extends the concepts involved with previous algorithms to enable faster image processing and a greater scope for using the inherent parallelism of the neural network approach to image processing. The simulation in this investigation shows that the new algorithm is able to maximize the efficiency of the described partitioning methods. This paper also presents an example of an application of the proposed algorithm to restore images degraded by motion blur.
---
paper_title: Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation.

paper_content:

The application of regularization to ill-conditioned problems necessitates the choice of a regularization parameter which trades fidelity to the data with smoothness of the solution. The value of the regularization parameter depends on the variance of the noise in the data. The problem of choosing the regularization parameter and estimating the noise variance in image restoration is examined. An error analysis based on an objective mean-square-error (MSE) criterion is used to motivate regularization. Two approaches for choosing the regularization parameter and estimating the noise variance are proposed. The proposed and existing methods are compared and their relationship to linear minimum-mean-square-error filtering is examined. Experiments are presented that verify the theoretical results.
---
paper_title: Support vector method for robust ARMA system identification

paper_content:

This paper presents a new approach to auto-regressive and moving average (ARMA) modeling based on the support vector method (SVM) for identification applications. A statistical analysis of the characteristics of the proposed method is carried out. An analytical relationship between residuals and SVM-ARMA coefficients allows the linking of the fundamentals of SVM with several classical system identification methods. Additionally, the effect of outliers can be cancelled. Application examples show the performance of SVM-ARMA algorithm when it is compared with other system identification methods.
---
paper_title: A least-squares based method for autoregressive signals in the presence of noise

paper_content:

The problem of estimating the unknown parameters of an autoregressive (AR) signal observed in white noise, including the signal power and the noise variance, is studied. A new type of least-squares method is developed which is based on a simple technique of estimating the observation noise variance by increasing the degree of the underlying AR model by one. The main feature of the presented method is that the consistent estimates of AR parameters can be directly achieved, with no need to prefilter noisy data or to make any parameter transformation.
---
paper_title: A Dual Neural Network for Redundancy Resolution of Kinematically Redundant Manipulators Subject to Joint Limits and Joint Velocity Limits

paper_content:

In this paper, a recurrent neural network called the dual neural network is proposed for online redundancy resolution of kinematically redundant manipulators. Physical constraints such as joint limits and joint velocity limits, together with the drift-free criterion as a secondary task, are incorporated into the problem formulation of redundancy resolution. Compared to other recurrent neural networks, the dual neural network is piecewise linear and has much simpler architecture with only one layer of neurons. The dual neural network is shown to be globally (exponentially) convergent to optimal solutions. The dual neural network is simulated to control the PA10 robot manipulator with effectiveness demonstrated.
---
paper_title: Fast recursive estimation of the parameters of a space-varying autoregressive image model

paper_content:

The identification of two-dimensional (2-D) autoregressive (AR) image models has been previously shown to be an integral part of image estimation. Furthermore, because of the nonhomogeneous nature of images, much better results are obtained with space varying models. To this effect, the development of a fast recursive method is now proposed for estimating the parameters of a two-dimensional AR image model, at each pixel, based on a finite memory. This fast method can be coupled to a space-variant Kalman filter for on-line adaptive estimation or can be used to estimate 2-D spectra for space-variant fields.
---
paper_title: A Recursive Least Squares Implementation for LCMP Beamforming Under Quadratic Constraint

paper_content:

Quadratic constraints on the weight vector of an adaptive linearly constrained minimum power (LCMP) beamformer can improve robustness to pointing errors and to random perturbations in sensor parameters. We propose a technique for implementing a quadratic inequality constraint with recursive least squares (RLS) updating. A variable diagonal loading term is added at each step, where the amount of loading has a closed-form solution. Simulations under different scenarios demonstrate that this algorithm has better interference suppression than both the RLS beamformer with no quadratic constraint and the RLS beamformer using the scaled projection technique, as well as faster convergence than LMS beamformers.
---
paper_title: Rapid Convergence Rate in Adaptive Arrays

paper_content:

In many applications, the practical usefulness of adaptive arrays is limited by their convergence rate. The adaptively controlled weights in these systems must change at a rate equal to or greater than the rate of change of the external noise field (e.g., due to scanning in a radar if step scan is not used). This convergence rate problem is most severe in adaptive systems with a large number of degrees of adaptivity and in situations where the eigenvalues of the noise covariance matrix are widely different. A direct method of adaptive weight computation, based on a sample covariance matrix of the noise field, has been found to provide very rapid convergence in all cases, i.e., independent of the eigenvalue distribution. A theory has been developed, based on earlier work by Goodman, which predicts the achievable convergence rate with this technique, and has been verified by simulation.
---
paper_title: A neural network for robust LCMP beamforming

paper_content:

Calculating an optimal beamforming weight is a main task of beamforming. Quadratic constraints on the weight vector of an adaptive linearly constrained minimum power (LCMP) beamformer can improve robustness to pointing errors and to random perturbations in sensor parameters. This paper presents a neural network approach to the robust LCMP beamformer with the quadratic constraint. Compared with the existing neural networks for the LCMP beamformer, the proposed neural network converges fast to an optimal weight. Compared with the existing adaptive algorithms for the robust LCMP beamformer, in addition to parallel implementation, the proposed neural network is guaranteed to converge exponentially to an optimal weight. Simulations demonstrate that the proposed neural network has better interference suppression and faster convergence than the existing neural networks and the adaptive algorithms.
---
paper_title: Neural methods for antenna array signal processing: a review

paper_content:

The neural method is a powerful nonlinear adaptive approach in various signal-processing scenarios. It is especially suitable for real-time application and hardware implementation. In this paper, we review its application in antenna array signal processing. This paper also serves as a tutorial to the neural method for antenna array signal processing.
---
paper_title: A Neural Minor Component Analysis Approach to Robust Constrained Beamforming

paper_content:

Since the pioneering work of Amari and Oja, principal component neural networks and their extensions have become an active adaptive signal processing research field. One of such extensions is minor component analysis (MCA), that proves to be effective in tasks such as robust curve/surface fitting and noise reduction. The aims of the paper are to give a detailed and homogeneous review of one-unit first minor/principal component analysis and to propose an application to robust constrained beamforming. In particular, after a careful presentation of first/minor component analysis algorithms based on a single adaptive neuron, along with relevant convergence/steady-state theorems, it is shown how the adaptive robust constrained beamforming theory by Cox et al. may be advantageously recast into an MCA setting. Experimental results obtained with a triangular array of microphones introduced in a teleconference context help to assess the usefulness of the proposed theory.
---
paper_title: A neural network approach to MVDR beamforming problem

paper_content:

A Hopfield-type neural network approach which leads to an analog circuit for implementing the real-time adaptive antenna array is presented. An optimal array pattern can be steered by updating the weights across the array in order to maximize the output signal-to-noise ratio (SNR). The problem of adjusting the array weights can be characterized as a constrained quadratic nonlinear programming. The adjustment of settings is required to respond to a rapid time-varying environment. A Hopfield-type neural net with a number of graded-response neurons designed to perform the constrained quadratic nonlinear programming would lead to a solution in a time determined by RC time constants, not by algorithmic time complexity. The constrained quadratic programming neural net has associated it with an energy function which the net always seeks to minimize. A fourth-order Runge-Kutta simulation shows that the circuit operates at a much higher speed than conventional techniques and the computation time of solving a linear array of 10 elements is about 0.1 ns for RC=5*10/sup -9/. >
---
paper_title: System identification using modular neural network with improved learning

paper_content:

This paper addresses the problem of the identification of nonlinear dynamic systems using modularly structured neural network with the new learning algorithm for the learning of both gating and expert networks weights. Here we start with the standard learning procedure for such networks in the sense that the problem of learning is formulated and treated as a mixture estimation problem in which the log-likelihood function should be maximised. But, as opposite to the established methods for modular networks we combine the gradient type of learning for gating weights with a least-squares algorithm for the learning of expert networks weights, and the experts are simple one-layer nets with a single linear output unit, The very result of such an approach is a simple structured modular network with improved learning and it seems with good capabilities for the identification of general nonlinear dynamic systems. The identifications of a discrete-time nonlinear system corrupted with noise and of a real world system are presented. Later process represents the identification of the positioning of a car engine throttle valve from real data set (3000 samples of measured noisy data).
---
paper_title: Comparison of Some Instrumental Variable Methods-Consistency and Accuracy Aspects

paper_content:

Abstract A general analysis of various instrumental variable methods is given. The lnvestigation includes methods for open loop operation as well as methods designed for closed loop operation. The instrumental variables are formed as different combinations of inputs, delayed inputs, delayed outputs, filtered inputs and external setpoint variations. Necessary conditions as well as sufficient conditions for consistency are explicitly derived and discussed. The paper also includes some simple counter-examples to general consistency for some common instrumental variable methods. Also asymptotic accuracy properties are reviewed. It is e.g. studied how the use of filters will influence the covariance matrix of the parameter estimates.
---
paper_title: A new neural network for solving linear programming problems and its application

paper_content:

Presents a new neural network which improves existing neural networks for solving general linear programming problems. The network, without setting parameter, uses only simple hardware in which no analog multipliers are required, and is proved to be completely stable to the exact solutions. Moreover, using this network the author can solve linear programming problems and its dual simultaneously, and cope with problems with nonunique solutions whose set is allowed to be unbounded.
---
paper_title: Cumulant-based order determination of non-Gaussian ARMA models

paper_content:

The objective is the order determination of non-Gaussian and nonminimum phase autoregressive moving average (ARMA) models, using higher-order cumulant statistics. The two methods developed assume knowledge of upper bounds on the ARMA orders. The first method performs a linear dependence search among the columns of a higher-order statistics matrix by means of the Gram-Schmidt orthogonalization procedure. In the second method, the order of the AR part is found as the rank of the matrix formed by the higher-order statistics sequence. For numerically robust rank determination the singular value decomposition approach is adopted. The argument principle and samples of the polyspectral phase are used to obtain the relative degree of the ARMA model, from which the order of the MA part can be determined. Statistical analysis is included for determining the correct MA order with high probability, when estimates of third-order cumulants are only available. Simulations are used to verify the performance of the methods and compare autocorrelation with cumulant-based order determination approaches. >
---
paper_title: A least-squares based method for autoregressive signals in the presence of noise

paper_content:

The problem of estimating the unknown parameters of an autoregressive (AR) signal observed in white noise, including the signal power and the noise variance, is studied. A new type of least-squares method is developed which is based on a simple technique of estimating the observation noise variance by increasing the degree of the underlying AR model by one. The main feature of the presented method is that the consistent estimates of AR parameters can be directly achieved, with no need to prefilter noisy data or to make any parameter transformation.
---
paper_title: Adaptive algorithms for constrained ARMA signals in the presence of noise

paper_content:

A new family of algorithms is developed for adaptive parameter estimation of constrained autoregressive moving-average (ARMA) signals in the presence of noise. These algorithms utilize a priori known information concerning the signal's properties, such as its spectral shape or a spatial domain characteristic. Special cases include autoregressive (AR) and band-pass spectrum signals in the presence of noise, signal deconvolution and image deblurring.
---
paper_title: A Generalized Least Absolute Deviation Method for Parameter Estimation of Autoregressive Signals

paper_content:

This paper proposes a generalized least absolute deviation (GLAD) method for parameter estimation of autoregressive (AR) signals under non-Gaussian noise environments. The proposed GLAD method can improve the accuracy of the estimation of the conventional least absolute deviation (LAD) method by minimizing a new cost function with parameter variables and noise error variables. Compared with second- and high-order statistical methods, the proposed GLAD method can obtain robustly an optimal AR parameter estimation without requiring the measurement noise to be Gaussian. Moreover, the proposed GLAD method can be implemented by a cooperative neural network (NN) which is shown to converge globally to the optimal AR parameter estimation within a finite time. Simulation results show that the proposed GLAD method can obtain more accurate estimates than several well-known estimation methods in the presence of different noise distributions.
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: Nonlinear system identification via direct weight optimization

paper_content:

A general framework for estimating nonlinear functions and systems is described and analyzed in this paper. Identification of a system is seen as estimation of a predictor function. The considered predictor function estimate at a particular point is defined to be affine in the observed outputs and the estimate is defined by the weights in this expression. For each given point, the maximal mean-square error (or an upper bound) of the function estimate over a class of possible true functions is minimized with respect to the weights, which is a convex optimization problem. This gives different types of algorithms depending on the chosen function class. It is shown how the classical linear least squares is obtained as a special case and how unknown-but-bounded disturbances can be handled. Most of the paper deals with the method applied to locally smooth predictor functions. It is shown how this leads to local estimators with a finite bandwidth, meaning that only observations in a neighborhood of the target point will be used in the estimate. The size of this neighborhood (the bandwidth) is automatically computed and reflects the noise level in the data and the smoothness priors. The approach is applied to a number of dynamical systems to illustrate its potential.
---
paper_title: A Function Estimation Approach to Sequential Learning with Neural Networks

paper_content:

In this paper, we investigate the problem of optimal sequential learning, viewed as a problem of estimating an underlying function sequentially rather than estimating a set of parameters of the neural network. First, we arrive at a suboptimal solution to the sequential estimate that can be mapped by a growing gaussian radial basis function (GaRBF) network. This network adds hidden units for each observation. The function space approach in which the estimates are represented as vectors in a function space is used in developing a growth criterion to limit its growth. A simplification of the criterion leads to two joint criteria on the distance of the present pattern and the existing unit centers in the input space and on the approximation error of the network for the given observation to be satisfied together. This network is similar to the resource allocating network (RAN) (Platt 1991a) and hence RAN can be interpreted from a function space approach to sequential learning. Second, we present an enhancement to the RAN. The RAN either allocates a new unit based on the novelty of an observation or adapts the network parameters by the LMS algorithm. The function space interpretation of the RAN lends itself to an enhancement of the RAN in which the extended Kalman filter (EKF) algorithm is used in place of the LMS algorithm. The performance of the RAN and the enhanced network are compared in the experimental tasks of function approximation and time-series prediction demonstrating the superior performance of the enhanced network with fewer number of hidden units. The approach adopted here has led us toward the minimal network required for a sequential learning problem.
---
paper_title: Second-order Volterra system identification

paper_content:

Second-order Volterra system identification is discussed. Crosscumulant information is converted into a Fredholm integral equation. Special emphasis is focused on inputs obtained at the output of a linear filter driven by higher order white noise.
---
paper_title: Neural network realization of support vector methods for pattern classification

paper_content:

We apply a recurrent neural network to support vector machine (SVM) training for pattern recognition. Specifically, a primal-dual neural network is exploited to solve the quadratic programming problem encountered in training SVMs. The properties of the network allow one to design SVMs without adjustable network parameters and give a better solution for ill-posed problems.
---
paper_title: Neural networks for nonlinear dynamic system modelling and identification

paper_content:

Many real-world systems exhibit complex nonlinear characteristics and cannot be treated satisfactorily using linear systems theory. A neural network which has the ability to learn sophisticated nonlinear relationships provides an ideal means of modelling complicated nonlinear systems. This paper addresses the issues related to the identification of nonlinear discrete-time dynamic systems using neural networks. Three network architectures, namely the multi-layer perceptron, the radial basis function network and the functional-link network, are presented and several learning or identification algorithms are derived. Advantages and disadvantages of these structures are discussed and illustrated using simulated and real data. Particular attention is given to the connections between existing techniques for nonlinear systems identification and some aspects of neural network methodology, and this demonstrates that certain techniques employed in the neural network context have long been developed by the control e...
---
paper_title: Predicting time series with support vector machines

paper_content:

Support Vector Machines are used for time series prediction and compared to radial basis function networks. We make use of two different cost functions for Support Vectors: training with (i) an e insensitive loss and (ii) Huber's robust loss function and discuss how to choose the regularization parameters in these models. Two applications are considered: data from (a) a noisy (normal and uniform noise) Mackey Glass equation and (b) the Santa Fe competition (set D). In both cases Support Vector Machines show an excellent performance. In case (b) the Support Vector approach improves the best known result on the benchmark by a factor of 29%.
---
paper_title: Nonlinear system identification via direct weight optimization

paper_content:

A general framework for estimating nonlinear functions and systems is described and analyzed in this paper. Identification of a system is seen as estimation of a predictor function. The considered predictor function estimate at a particular point is defined to be affine in the observed outputs and the estimate is defined by the weights in this expression. For each given point, the maximal mean-square error (or an upper bound) of the function estimate over a class of possible true functions is minimized with respect to the weights, which is a convex optimization problem. This gives different types of algorithms depending on the chosen function class. It is shown how the classical linear least squares is obtained as a special case and how unknown-but-bounded disturbances can be handled. Most of the paper deals with the method applied to locally smooth predictor functions. It is shown how this leads to local estimators with a finite bandwidth, meaning that only observations in a neighborhood of the target point will be used in the estimate. The size of this neighborhood (the bandwidth) is automatically computed and reflects the noise level in the data and the smoothness priors. The approach is applied to a number of dynamical systems to illustrate its potential.
---
paper_title: Description of the OHGR Database.

paper_content:

Abstract : This Technical Note describes the Osborne Head database, collected at Osborne Head Gunnery Range, OHGR, in November 1993, with the McMaster University IPIX radar, under contract to DSS. Representative examples are worked out for the different operational modes of the radar (staring, scanning, alternate/single polarization, single/multi frequency). Strengths and weaknesses of the database are pointed out as well. The purpose of this Technical Note is to serve as a guide to the database, presenting enough information to allow the extraction of individual datasets from the raw data. These can subsequently be used for sea clutter, target model validation and/or testing of signal processing techniques, leading to enhanced target detection in sea clutter.
---
paper_title: Signal detection using the radial basis function coupled map lattice

paper_content:

From observation sea clutter, radar echoes from a sea surface, is chaotic rather than random. We propose the use of a spatial temporal predictor to reconstruct the chaotic dynamic of sea clutter because electromagnetic wave scattering is a spatial temporal phenomenon which is physically modeled by partial differential equations. The spatial temporal predictor used here is called radial basis function coupled map lattice (RBF-CML) which uses linear combination to fuse either measurements in different spatial domains for an RBF prediction or predictions from several RBF nets operated on different spatial regions. Using real-life radar data, it is shown that the RBF-CML is an effective method to reconstruct the sea clutter dynamic. The RBF-CML predictor is then applied to detect small targets in sea clutter using the constant false alarm rate (CFAR) principle. The spatial temporal approach is shown, both theoretically and experimentally, to be superior to a conventional CFAR detector.
---
paper_title: A fully neural-network-based planning scheme for torque minimization of redundant manipulators

paper_content:

The aim of this paper is to develop a new method for minimizing joint torques of redundant manipulators in the Chebyshev sense and to present a fully neural-network-based computational scheme for its implementation. Minimax techniques are used to determine the null space acceleration vector which can guarantee to minimize the maximum joint torque. For real-time implementation, we transform the proposed method into a computation of a recurrent neural network. At each time step, the neural network is adopted for both the solution of the least-norm joint acceleration and the determination of the optimum null space acceleration vector. Compared with previous torque minimization schemes, the proposed method enables more direct monitoring and control of the magnitudes of the individual joint torques than does the minimization of the sum of squares of the components. Simulation results demonstrate that the proposed method is effective for the torque minimization control of redundant manipulators.
---
paper_title: A Lagrangian network for kinematic control of redundant robot manipulators

paper_content:

A recurrent neural network, called the Lagrangian network, is presented for the kinematic control of redundant robot manipulators. The optimal redundancy resolution is determined by the Lagrangian network through real-time solution to the inverse kinematics problem formulated as a quadratic optimization problem. While the signal for a desired velocity of the end-effector is fed into the inputs of the Lagrangian network, it generates the joint velocity vector of the manipulator in its outputs along with the associated Lagrange multipliers. The proposed Lagrangian network is shown to be capable of asymptotic tracking for the motion control of kinematically redundant manipulators.
---
paper_title: Motion control of 7-DOF arms: the configuration control approach

paper_content:

Graphics simulation and real-time implementation of configuration control schemes for a redundant seven-degree-of-freedom (7-DOF) Robotics Research arm are described. The arm kinematics and motion control schemes are described briefly. This is followed by a description of a graphics simulation environment for 7-DOF arm control on the Silicon Graphics IRIS Workstation. Computer simulation results are presented to demonstrate elbow control, collision avoidance, and optimal joint movement as redundancy resolution goals. The laboratory setup for experimental validation of motion control of the 7-DOF Robotics Research arm is then described. The configuration control approach is implemented on a Motorola-68020/VME-bus-based real-time controller, with elbow positioning for redundancy resolution. Experimental results demonstrate the efficacy of configuration control for real-time control. >
---
paper_title: Accelerated convergence in the inverse kinematics via multilayer feedforward networks

paper_content:

A hybrid approach to the iterative solution of robotic manipulators is presented. In this method a multiplayer feedforward network (MFN) is trained to provide an approximate solution to the inverse kinematic problem of a robot. This approximate solution is then utilized as the initial guess to the iterative procedure which provides the solution within some specified tolerance. Examples involving the PUMA 560 robot are given. Since the time taken by the MFN to provide the initial estimate is only a fraction of the time taken by the iterative procedure taken to reach a solution, this combination can be usefully employed in order to decrease the time for obtaining the solution to the inverse kinematic problem in robotics. >
---
paper_title: Obstacle Avoidance for Kinematically Redundant Manipulators in Dynamically Varying Environments

paper_content:

The vast majority of work to date concerned with obstacle avoidance for manipulators has dealt with task descriptions in the form ofpick-and-place movements. The added flexibil ity in motion control for manipulators possessing redundant degrees offreedom permits the consideration of obstacle avoidance in the context of a specified end-effector trajectory as the task description. Such a task definition is a more accurate model for such tasks as spray painting or arc weld ing. The approach presented here is to determine the re quired joint angle rates for the manipulator under the con straints of multiple goals, the primary goal described by the specified end-effector trajectory and secondary goals describ ing the obstacle avoidance criteria. The decomposition of the solution into a particular and a homogeneous component effectively illustrates the priority of the multiple goals that is exact end-effector control with redundant degrees of freedom maximizing the distance to obstacles. An efficient numerical ...
---
paper_title: Resolving manipulator redundancy under inequality constraints

paper_content:

Due to hardware limitations, physical constraints such as joint rate bounds, joint angle limits, and joint torque constraints always exist. In this paper, these constraints are considered in the general formulation of the redundant inverse kinematic problem. To take these physical constraints into account, the computationally efficient compact quadratic programming (QP) method is formed to resolve the constrained kinematic redundancy problem. In addition, the compact-inverse QP method is also formulated to remedy the inescapable singularity problem with inequality constraints. Two examples are given to demonstrate the generality and superiority of these two methods: to eliminate the drift phenomenon caused by self motion and to remedy saturation-type nonlinearity problems. >
---

Title: Cooperative Recurrent Modular Neural Networks for Constrained Optimization: A Survey of Models and Applications

Section 1: Introduction
Description 1: Write about the advancements in single neural networks and the inception of modular and cooperative neural network approaches. Discuss the limitations of single neural networks in solving complex problems and the emergence of cooperative recurrent neural networks for constrained optimization.

Section 2: Modular Neural Networks and Cooperative Modular Neural Networks
Description 2: Describe the basic structure and motivation behind modular neural networks and how cooperative modular neural networks differ from traditional modular neural networks. Include their advantages and applications.

Section 3: Ensemble Neural Networks
Description 3: Explain the concept of neural network ensembles, their structure, and how they are used to improve the performance of single neural networks. Discuss their use in various applications.

Section 4: Cooperative Recurrent Neural Networks for Constrained Optimization
Description 4: Provide an in-depth survey of cooperative recurrent neural networks for constrained optimization, including their models and mathematical formulations. 

Section 5: Problem Formulation and Single Neural Networks
Description 5: Discuss the problem formulation for constrained optimization and the initial approaches using single neural networks. Include the limitations of single neural network models.

Section 6: Survey of Cooperative Recurrent Neural Network Models
Description 6: Survey various models of cooperative recurrent neural networks, their theoretical foundations, and algorithmic details. Highlight the advantages and convergence properties of these models.

Section 7: Cooperative Recurrent Neural Networks for the LAD Estimation
Description 7: Detail the application of cooperative recurrent neural networks in solving least absolute deviation (LAD) problems. Discuss the model formulations and convergence properties.

Section 8: Survey of Applications of CRNNs
Description 8: Explore the diverse applications of cooperative recurrent neural networks, including classification, data fusion, image restoration, and nonlinear system identification. Discuss how CRNNs are applied and their performance in different domains.

Section 9: CRNNs for Classification
Description 9: Describe the CRNN approach for solving classification problems, including problem formulations and real-time classification scenarios. Provide examples and experimental results.

Section 10: CRNNs for Data Fusion
Description 10: Discuss how CRNNs are used for data fusion in various applications. Explain the methodologies and provide case studies or examples.

Section 11: CRNNs for Image Restoration
Description 11: Explain the application of CRNNs in image restoration problems. Include algorithm details, examples, and performance comparisons with other methods.

Section 12: CRNNs for Parameter Estimation of AR Signals 
Description 12: Detail the use of CRNNs for estimating autoregressive (AR) signal parameters. Discuss the problem formulation, algorithm, and performance results.

Section 13: Nonlinear System Identification
Description 13: Explain how CRNNs are applied to the nonlinear system identification. Discuss the methodologies and provide examples of success in real-world applications.

Section 14: CRNNs for On-line Resolution of Constrained Kinematic Redundancy
Description 14: Discuss the application of CRNNs to solve on-line kinematic redundancy resolution problems in robotics. Provide algorithm details and examples.

Section 15: Conclusion
Description 15: Summarize the survey findings, highlighting the advantages of cooperative recurrent neural networks over other approaches for constrained optimization. Discuss open problems and future research directions.