A survey on universal approximation and its limits in soft computing techniques
---
paper_title: Implications and applications of Kolmogorov's superposition theorem

paper_content:

Applications of Kolmogorov's superposition theorem to non-linear circuit and system theory, statistical pattern recognition, and image and multidimensional signal processing are presented and discussed.
---
paper_title: Fuzzy systems are universal approximators

paper_content:

The author proves that fuzzy systems are universal approximators. The Stone-Weierstrass theorem is used to prove that fuzzy systems with product inference, centroid defuzzification, and a Gaussian membership function are capable of approximating any real continuous function on a compact set to arbitrary accuracy. This result can be viewed as an existence theorem of an optimal fuzzy system for a wide variety of problems. >
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: Kolmogorov's theorem and multilayer neural networks

paper_content:

Abstract Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof, we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation.
---
paper_title: Sufficient conditions on uniform approximation of multivariate functions by general Takagi-Sugeno fuzzy systems with linear rule consequent

paper_content:

We have constructively proved a general class of multi-input single-output Takagi-Sugeno (TS) fuzzy systems to be universal approximators. The systems use any types of continuous fuzzy sets, fuzzy logic AND, fuzzy rules with linear rule consequent and the generalized defuzzifier. We first prove that the TS fuzzy systems can uniformly approximate any multivariate polynomial arbitrarily well, and then prove they can also uniformly approximate any multivariate continuous function arbitrarily well. We have derived a formula for computing the minimal upper bounds on the number of fuzzy sets and fuzzy rules necessary to achieve the prespecified approximation accuracy for any given bivariate function. A numerical example is furnished. Our results provide a solid-theoretical basis for fuzzy system applications, particularly as fuzzy controllers and models.
---
paper_title: Approximation theory and feedforward networks

paper_content:

Approximation of real functions by feedforward networks of the usual kind is shown to be based on the fundamental principle of approximation by piecewise-constant functions. This principle underlies a simple construction given for three-layer networks and suggests possible difficulties in determining two-layer networks.
---
paper_title: Necessary conditions on minimal system configuration for general MISO Mamdani fuzzy systems as universal approximators

paper_content:

Recent studies have shown that both Mamdani-type and Takagi-Sugeno-type fuzzy systems are universal approximators in that they can uniformly approximate continuous functions defined on compact domains with arbitrarily high approximation accuracy. In this paper, we investigate necessary conditions for general multiple-input single-output (MISO) Mamdani fuzzy systems as universal approximators with as minimal system configuration as possible. The general MISO fuzzy systems employ almost arbitrary continuous input fuzzy sets, arbitrary singleton output fuzzy sets, arbitrary fuzzy rules, product fuzzy logic AND, and the generalized defuzzifier containing the popular centroid defuzzifier as a special case. Our necessary conditions are developed under the practically sensible assumption that only a finite set of extrema of the multivariate continuous function to be approximated is available. We have first revealed a decomposition property of the general fuzzy systems: A r-input fuzzy system can always be decomposed to the sum of r simpler fuzzy systems where the first system has only one input variable, the second one two input variables, and the last one r input variables. Utilizing this property, we have derived some necessary conditions for the fuzzy systems to be universal approximators with minimal system configuration. The conditions expose the strength as well as limitation of the fuzzy approximation: (1) only a small number of fuzzy rules may be needed to uniformly approximate multivariate continuous functions that have a complicated formulation but a relatively small number of extrema; and (2) the number of fuzzy rules must be large in order to approximate highly oscillatory continuous functions. A numerical example is given to demonstrate our new results.
---
paper_title: Fuzzy logic controllers are universal approximators

paper_content:

In this paper, we consider a fundamental theoretical question on why does fuzzy control have such a good performance for a wide variety of practical problems. We try to answer this fundamental question by proving that for each fixed fuzzy logic belonging to a wide class of fuzzy logics, and for each fixed type of membership function belonging to a wide class of membership functions, the fuzzy logic control systems using these two and any method of defuzzification are capable of approximating any real continuous function on a compact set to arbitrary accuracy. On the other hand, this result can be viewed as an existence theorem of an optimal fuzzy logic control system for a wide variety of problems. >
---
paper_title: A comparative study on sufficient conditions for Takagi-Sugeno fuzzy systems as universal approximators

paper_content:

Universal approximation is the basis of theoretical research and practical applications of fuzzy systems. Studies on the universal approximation capability of fuzzy systems have achieved great progress in recent years. In this paper, linear Takagi-Sugeno (TS) fuzzy systems that use linear functions of input variables as rule consequent and their special case, named simplified fuzzy systems that use fuzzy singletons as rule consequent, are investigated. On condition that overlapped fuzzy sets are employed, new sufficient conditions for simplified fuzzy systems and linear TS fuzzy systems as universal approximators are given, respectively. Then, a comparative study on existing sufficient conditions is carried out with numeric examples.
---
paper_title: Fuzzy systems as universal approximators

paper_content:

An additive fuzzy system can uniformly approximate any real continuous function on a compact domain to any degree of accuracy. An additive fuzzy system approximates the function by covering its graph with fuzzy patches in the input-output state space and averaging patches that overlap. The fuzzy system computes a conditional expectation E|Y|X| if we view the fuzzy sets as random sets. Each fuzzy rule defines a fuzzy patch and connects commonsense knowledge with state-space geometry. Neural or statistical clustering systems can approximate the unknown fuzzy patches from training data. These adaptive fuzzy systems approximate a function at two levels. At the local level the neural system approximates and tunes the fuzzy rules. At the global level the rules or patches approximate the function. >
---
paper_title: ARE FUZZY SYSTEMS UNIVERSAL APPROXIMATORS?

paper_content:

Abstract This paper is a critical reflection on various results in lileralure claiming that fuzzy systems are universal approximators. For this purpose the most specific features of fuzzy systems are outlined and it is discussed to which extent they are incorporated in the formal definition of a Tuzzy system in literature. It is argued that fuzzy systems can only be universal approximators in a rather reduced sense where some crucial features are neglected. The goal is to give an impulse to investigate more adequate mathematical concepts of a fuzzy system thai also take into account such features as transparency and linguistic interpretability.
---
paper_title: Implications and applications of Kolmogorov's superposition theorem

paper_content:

Applications of Kolmogorov's superposition theorem to non-linear circuit and system theory, statistical pattern recognition, and image and multidimensional signal processing are presented and discussed.
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: Kolmogorov's theorem and multilayer neural networks

paper_content:

Abstract Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof, we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation.
---
paper_title: On the approximate realization of continuous mappings by neural networks

paper_content:

Abstract In this paper, we prove that any continuous mapping can be approximately realized by Rumelhart-Hinton-Williams' multilayer neural networks with at least one hidden layer whose output functions are sigmoid functions. The starting point of the proof for the one hidden layer case is an integral formula recently proposed by Irie-Miyake and from this, the general case (for any number of hidden layers) can be proved by induction. The two hidden layers case is proved also by using the Kolmogorov-Arnold-Sprecher theorem and this proof also gives non-trivial realizations.
---
paper_title: Approximation capabilities of multilayer feedforward networks

paper_content:

Abstract We show that standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to L p (μ) performance criteria, for arbitrary finite input environment measures μ, provided only that sufficiently many hidden units are available. If the activation function is continuous, bounded and nonconstant, then continuous mappings can be learned uniformly over compact input sets. We also give very general conditions ensuring that networks with sufficiently smooth activation functions are capable of arbitrarily accurate approximation to a function and its derivatives.
---
paper_title: Approximation theory and feedforward networks

paper_content:

Approximation of real functions by feedforward networks of the usual kind is shown to be based on the fundamental principle of approximation by piecewise-constant functions. This principle underlies a simple construction given for three-layer networks and suggests possible difficulties in determining two-layer networks.
---
paper_title: Approximation by superpositions of sigmoidal functions

paper_content:

In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set of affine functionals can uniformly approximate any continuous function ofn real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single hidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.
---
paper_title: Neural networks with a continuous squashing function in the output are universal approximators

paper_content:

In 1989 Hornik as well as Funahashi established that multilayer feedforward networks without the squashing function in the output layer are universal approximators. This result has been often used improperly because it has been applied to multilayer feedforward networks with the squashing function in the output layer. In this paper, we will prove that also this kind of neural networks are universal approximators, i.e. they are capable of approximating any Borel measurable function from one finite dimensional space into (0,1)" to any desired degree of accuracy, provided sufficiently many hidden units are available.
---
paper_title: Hierarchical fuzzy control

paper_content:

In this paper, a simplified method for developing Takagi-Sugeno-Kang (TSK) style fuzzy controllers is described. The method takes advantage of developer's knowledge and understanding of the system being implemented. System attributes such as variable and nonlinearity independence are utilized to significantly reduce the required rule-base size when describing the system, without compromising robustness and performance. An example of a tracking system is used to convey the process of implementing a hierarchical fuzzy controller.
---
paper_title: Fuzzy systems are universal approximators

paper_content:

The author proves that fuzzy systems are universal approximators. The Stone-Weierstrass theorem is used to prove that fuzzy systems with product inference, centroid defuzzification, and a Gaussian membership function are capable of approximating any real continuous function on a compact set to arbitrary accuracy. This result can be viewed as an existence theorem of an optimal fuzzy system for a wide variety of problems. >
---
paper_title: Stability of a new interpolation method

paper_content:

Aims to complete the analysis of an /spl alpha/-cut based interpolation technique originating from the KH interpolation. Our goal is to investigate its stability behaviour. As it was shown in Joo et al. (1997) and Tikk et al. (1999) the original KH interpolation is stable in the sense that if inputs change slightly the output does not change much, as well. The main result of this paper shows that this significant feature of the KH interpolation can be carried over for the proposed method. A possible generalization of the proposed method is also presented.
---
paper_title: A note on universal approximation by hierarchical fuzzy systems

paper_content:

This paper proves, in a constructive manner, that the general n-dimensional hierarchical fuzzy systems are universal approximators. It is an extension of the results in [L.X. Wang, Fuzzy Sets and Systems 93 (1998) 223–230]. An upper bound of approximation error is also given.
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: General SISO Takagi-Sugeno fuzzy systems with linear rule consequent are universal approximators

paper_content:

Takagi-Sugeno (TS) fuzzy systems have been employed as fuzzy controllers and fuzzy models in successfully solving difficult control and modeling problems in practice. Virtually all the TS fuzzy systems use linear rule consequent. At present, there exist no results (qualitative or quantitative) to answer the fundamentally important question that is especially critical to TS fuzzy systems as fuzzy controllers and models, "Are TS fuzzy systems with linear rule consequent universal approximators?" If the answer is yes, then how can they be constructed to achieve prespecified approximation accuracy and what are the sufficient renditions on systems configuration? In this paper, we provide answers to these questions for a general class of single-input single-output (SISO) fuzzy systems that use any type of continuous input fuzzy sets, TS fuzzy rules with linear consequent and a generalized defuzzifier containing the widely used centroid defuzzifier as a special case. We first constructively prove that this general class of SISO TS fuzzy systems can uniformly approximate any polynomial arbitrarily well and then prove, by utilizing the Weierstrass approximation theorem, that the general TS fuzzy systems can uniformly approximate any continuous function with arbitrarily high precision. Furthermore, we have derived a formula as part of sufficient conditions for the fuzzy approximation that can compute the minimal upper bound on the number of input fuzzy sets and rules needed for any given continuous function and prespecified approximation error bound, An illustrative numerical example is provided.
---
paper_title: Sufficient conditions on uniform approximation of multivariate functions by general Takagi-Sugeno fuzzy systems with linear rule consequent

paper_content:

We have constructively proved a general class of multi-input single-output Takagi-Sugeno (TS) fuzzy systems to be universal approximators. The systems use any types of continuous fuzzy sets, fuzzy logic AND, fuzzy rules with linear rule consequent and the generalized defuzzifier. We first prove that the TS fuzzy systems can uniformly approximate any multivariate polynomial arbitrarily well, and then prove they can also uniformly approximate any multivariate continuous function arbitrarily well. We have derived a formula for computing the minimal upper bounds on the number of fuzzy sets and fuzzy rules necessary to achieve the prespecified approximation accuracy for any given bivariate function. A numerical example is furnished. Our results provide a solid-theoretical basis for fuzzy system applications, particularly as fuzzy controllers and models.
---
paper_title: Universal approximation by hierarchical fuzzy systems

paper_content:

Abstract A serious problem limiting the applicability of standard fuzzy controllers is the rule-explosion problem; that is, the number of rules increases exponentially with the number of input variables to the fuzzy controller. A way to deal with this “curse of dimensionality” is to use the hierarchical fuzzy systems. A hierarchical fuzzy system consists of a number of hierarchically connected low-dimensional fuzzy systems. It can be shown that the number of rules in the hierarchical fuzzy system increases linearly with the number of input variables. In this paper, we prove that the hierarchical fuzzy systems are universal approximators; that is, they can approximate any nonlinear function on a compact set to arbitrary accuracy. Our proof is constructive, that is, we first construct a hierarchical fuzzy system in a step-by-step manner, then prove that the constructed fuzzy system satisfies an error bound, and finally show that the error bound can be made arbitrarily small.
---
paper_title: General Takagi-Sugeno fuzzy systems with simplified linear rule consequent are universal controllers, models and filters

paper_content:

Abstract Takagi-Sugeno (TS) fuzzy systems have successfully been employed, mainly in a trial-and-error manner, to solve many control and modeling problems; but their applications as signal filters remain to be fully explored. Compared to their nonfuzzy counterparts, TS fuzzy controllers and models are difficult to be efficiently constructed because there is a large number of design parameters in the rule consequent. The number grows dramatically with the increase of the number of input fuzzy sets and input variables. Furthermore, there exists little published result on relationship between TS fuzzy controllers/models/filters and their nonfuzzy counterparts. In this paper, we investigate, in relation to some popular nonfuzzy controllers, models and filters, analytical structure of a general class of multi-input single-output (MISO) TS fuzzy systems that use arbitrary fuzzy rules with our recently introduced simplified linear rule consequent. Other components of the fuzzy systems in this study are general: arbitrary continuous input fuzzy sets, any type of fuzzy logic AND and the generalized defuzzifier containing the widely used centroid defuzzifier as a special case. We prove that the general MISO TS fuzzy systems are: (1) nonlinear variable gain controllers when implemented as controllers, or (2) nonlinear time-varying auto-regressive with the extra input (ARX) models when implemented as models, or (3) nonlinear infinite impulse response (IIR) or finite impulse response (FIR) filters when implemented as filters. Furthermore, we constructively prove that the general TS fuzzy systems with the simplified linear rule consequent are universal approximators and can approximate any continuous function in closed domain arbitrarily well. The practical implication of these results is that these fuzzy systems, with much less design parameters, are always able to produce solutions to various control, modeling and filtering problems. We also establish sufficient conditions that can be used to calculate the number of input fuzzy sets and rules needed for achieving prespecified approximation accuracy.
---
paper_title: Universal approximation by hierarchical fuzzy system with constraints on the fuzzy rule

paper_content:

This paper presents a special hierarchical fuzzy system where the outputs of the previous layer are not used in the IF-parts, but used only in the THEN-parts of the fuzzy rules of the current layer.The proposed scheme can be shown to be a universal approximator to any continuous function on a compact set if complete fuzzy sets are used in the IF-parts of the fuzzy rules with singleton fuzzifier and center average defuzzifier.From the simulation of ball and beam control system, it is demonstrated that the proposed scheme approximates with good accuracy the model nonlinear controller with fewer fuzzy rules than the centralized fuzzy system and its control performance is comparable to that of the nonlinear controller.
---
paper_title: Fuzzy logic controllers are universal approximators

paper_content:

In this paper, we consider a fundamental theoretical question on why does fuzzy control have such a good performance for a wide variety of practical problems. We try to answer this fundamental question by proving that for each fixed fuzzy logic belonging to a wide class of fuzzy logics, and for each fixed type of membership function belonging to a wide class of membership functions, the fuzzy logic control systems using these two and any method of defuzzification are capable of approximating any real continuous function on a compact set to arbitrary accuracy. On the other hand, this result can be viewed as an existence theorem of an optimal fuzzy logic control system for a wide variety of problems. >
---
paper_title: Approximation theory of fuzzy systems based upon genuine many-valued implications - SISO cases

paper_content:

It is proved that the single input and single output (SISO) fuzzy systems based upon genuine many-valued implications are universal approximators. It is shown theoretically that fuzzy control systems based upon genuine many-valued implications are equivalent to those based upon t-norm implications, the general approach to construct fuzzy systems is given. It is also shown that defuzzifier based upon center of areas is not appropriate to the fuzzy systems based upon genuine many-valued implications.
---
paper_title: A comparative study on sufficient conditions for Takagi-Sugeno fuzzy systems as universal approximators

paper_content:

Universal approximation is the basis of theoretical research and practical applications of fuzzy systems. Studies on the universal approximation capability of fuzzy systems have achieved great progress in recent years. In this paper, linear Takagi-Sugeno (TS) fuzzy systems that use linear functions of input variables as rule consequent and their special case, named simplified fuzzy systems that use fuzzy singletons as rule consequent, are investigated. On condition that overlapped fuzzy sets are employed, new sufficient conditions for simplified fuzzy systems and linear TS fuzzy systems as universal approximators are given, respectively. Then, a comparative study on existing sufficient conditions is carried out with numeric examples.
---
paper_title: Fuzzy systems as universal approximators

paper_content:

An additive fuzzy system can uniformly approximate any real continuous function on a compact domain to any degree of accuracy. An additive fuzzy system approximates the function by covering its graph with fuzzy patches in the input-output state space and averaging patches that overlap. The fuzzy system computes a conditional expectation E|Y|X| if we view the fuzzy sets as random sets. Each fuzzy rule defines a fuzzy patch and connects commonsense knowledge with state-space geometry. Neural or statistical clustering systems can approximate the unknown fuzzy patches from training data. These adaptive fuzzy systems approximate a function at two levels. At the local level the neural system approximates and tunes the fuzzy rules. At the global level the rules or patches approximate the function. >
---
paper_title: Approximation by neural networks is not continuous

paper_content:

Abstract It is shown that in a Banach space X satisfying mild conditions, for its infinite, linearly independent subset G , there is no continuous best approximation map from X to the n -span, span n G . The hypotheses are satisfied when X is an L p -space, 1 p G is the set of functions computed by the hidden units of a typical neural network (e.g., Gaussian, Heaviside or hyperbolic tangent). If G is finite and span n G is not a subspace of X , it is also shown that there is no continuous map from X to span n G within any positive constant of a best approximation.
---
paper_title: Approximation of functions by perceptron networks with bounded number of hidden units

paper_content:

Abstract We examine the effect of constraining the number of hidden units. For one-hidden-layer networks with a fairly general type of units (including perceptrons with any bounded activation function and radial-basis-function units), we show that when also the size of parameters is bounded, the best approximation property is satisfied, which means that there always exists a parametrization achieving the global minimum of any error function generated by a supremum orLp-norm. We also show that the only functions that can be approximated with arbitrary accuracy by increasing parameters in networks with a fixed number of Heaviside perceptrons are functions equal almost everywhere to functions that can be exactly computed by such networks. We give a necessary condition on values that such piecewise constant functions must achieve.
---
paper_title: Polytopic and TS models are nowhere dense in the approximation model space

paper_content:

We show in this paper that the set of functions, consisting of polytopic or TS models constructed from finite number of components, is nowhere dense in the approximation model space, if that is defined as a subset of continuous functions. This topological notion means that the given set of functions lies "almost discretely" in the space of approximated functions. As a consequence, by means of the mentioned models we cannot approximate in general continuous functions arbitrarily well, if the number of components are restricted. Thus, only functions satisfying certain conditions can be approximated by such models, or alternatively, we need unbounded number of components. The possible solutions are outlined in the paper.
---
paper_title: Approximation capabilities of multilayer feedforward networks

paper_content:

Abstract We show that standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to L p (μ) performance criteria, for arbitrary finite input environment measures μ, provided only that sufficiently many hidden units are available. If the activation function is continuous, bounded and nonconstant, then continuous mappings can be learned uniformly over compact input sets. We also give very general conditions ensuring that networks with sufficiently smooth activation functions are capable of arbitrarily accurate approximation to a function and its derivatives.
---
paper_title: ARE FUZZY SYSTEMS UNIVERSAL APPROXIMATORS?

paper_content:

Abstract This paper is a critical reflection on various results in lileralure claiming that fuzzy systems are universal approximators. For this purpose the most specific features of fuzzy systems are outlined and it is discussed to which extent they are incorporated in the formal definition of a Tuzzy system in literature. It is argued that fuzzy systems can only be universal approximators in a rather reduced sense where some crucial features are neglected. The goal is to give an impulse to investigate more adequate mathematical concepts of a fuzzy system thai also take into account such features as transparency and linguistic interpretability.
---
paper_title: Universal approximation bounds for superpositions of a sigmoidal function

paper_content:

Approximation properties of a class of artificial neural networks are established. It is shown that feedforward networks with one layer of sigmoidal nonlinearities achieve integrated squared error of order O(1/n), where n is the number of nodes. The approximated function is assumed to have a bound on the first moment of the magnitude distribution of the Fourier transform. The nonlinear parameters associated with the sigmoidal nodes, as well as the parameters of linear combination, are adjusted in the approximation. In contrast, it is shown that for series expansions with n terms, in which only the parameters of linear combination are adjusted, the integrated squared approximation error cannot be made smaller than order 1/n/sup 2/d/ uniformly for functions satisfying the same smoothness assumption, where d is the dimension of the input to the function. For the class of functions examined, the approximation rate and the parsimony of the parameterization of the networks are shown to be advantageous in high-dimensional settings. >
---
paper_title: A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training

paper_content:

A general convergence criterion for certain iterative sequences in Hilbert space is presented. For an important subclass of these sequences, estimates of the rate of convergence are given. Under very mild assumptions these results establish an 0(1/ F4n) nonsampling convergence rate for projection pursuit regression and neural network training; where n represents the number of ridge functions, neurons or coefficients in a greedy basis expansion.
---
paper_title: Convergence rates for single hidden layer feedforward networks

paper_content:

Abstract By allowing the training set to become arbitrarily large, appropriately trained and configured single hidden layer feedforward networks converge in probability to the smooth function that they were trained to estimate. A bound on the probabilistic rate of convergence of these network estimates is given. The convergence rate is calculated as a function of the sample size n. If the function being estimated has square integrable m th order partial derivatives then the L 2 -norm estimation error approaches O p ( n − 1 2 ) for large m. Two steps are required for determining these bounds. A bound on the rate of convergence of approximations to an unknown smooth function by members of a special class of single hidden layer feedforward networks is determined. The class of networks considered can embed Fourier series. Using this fact and results on approximation properties of Fourier series yields a bound on L 2 -norm approximation error. This bound is less than O ( q − 1 2 ) for approximating a smooth function by networks with q hidden units. A modification of existing results for bounding estimation error provides a general theorem for calculating estimation error convergence rates. Combining this result with the bound on approximation rates yields the final convergence rates.
---
paper_title: Neural networks with a continuous squashing function in the output are universal approximators

paper_content:

In 1989 Hornik as well as Funahashi established that multilayer feedforward networks without the squashing function in the output layer are universal approximators. This result has been often used improperly because it has been applied to multilayer feedforward networks with the squashing function in the output layer. In this paper, we will prove that also this kind of neural networks are universal approximators, i.e. they are capable of approximating any Borel measurable function from one finite dimensional space into (0,1)" to any desired degree of accuracy, provided sufficiently many hidden units are available.
---
paper_title: Kolmogorov's theorem and multilayer neural networks

paper_content:

Abstract Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof, we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation.
---
paper_title: Approximation theory and feedforward networks

paper_content:

Approximation of real functions by feedforward networks of the usual kind is shown to be based on the fundamental principle of approximation by piecewise-constant functions. This principle underlies a simple construction given for three-layer networks and suggests possible difficulties in determining two-layer networks.
---
paper_title: Kolmogorov's theorem and multilayer neural networks

paper_content:

Abstract Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof, we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation.
---
paper_title: On the approximate realization of continuous mappings by neural networks

paper_content:

Abstract In this paper, we prove that any continuous mapping can be approximately realized by Rumelhart-Hinton-Williams' multilayer neural networks with at least one hidden layer whose output functions are sigmoid functions. The starting point of the proof for the one hidden layer case is an integral formula recently proposed by Irie-Miyake and from this, the general case (for any number of hidden layers) can be proved by induction. The two hidden layers case is proved also by using the Kolmogorov-Arnold-Sprecher theorem and this proof also gives non-trivial realizations.
---
paper_title: General SISO Takagi-Sugeno fuzzy systems with linear rule consequent are universal approximators

paper_content:

Takagi-Sugeno (TS) fuzzy systems have been employed as fuzzy controllers and fuzzy models in successfully solving difficult control and modeling problems in practice. Virtually all the TS fuzzy systems use linear rule consequent. At present, there exist no results (qualitative or quantitative) to answer the fundamentally important question that is especially critical to TS fuzzy systems as fuzzy controllers and models, "Are TS fuzzy systems with linear rule consequent universal approximators?" If the answer is yes, then how can they be constructed to achieve prespecified approximation accuracy and what are the sufficient renditions on systems configuration? In this paper, we provide answers to these questions for a general class of single-input single-output (SISO) fuzzy systems that use any type of continuous input fuzzy sets, TS fuzzy rules with linear consequent and a generalized defuzzifier containing the widely used centroid defuzzifier as a special case. We first constructively prove that this general class of SISO TS fuzzy systems can uniformly approximate any polynomial arbitrarily well and then prove, by utilizing the Weierstrass approximation theorem, that the general TS fuzzy systems can uniformly approximate any continuous function with arbitrarily high precision. Furthermore, we have derived a formula as part of sufficient conditions for the fuzzy approximation that can compute the minimal upper bound on the number of input fuzzy sets and rules needed for any given continuous function and prespecified approximation error bound, An illustrative numerical example is provided.
---
paper_title: Sufficient conditions on uniform approximation of multivariate functions by general Takagi-Sugeno fuzzy systems with linear rule consequent

paper_content:

We have constructively proved a general class of multi-input single-output Takagi-Sugeno (TS) fuzzy systems to be universal approximators. The systems use any types of continuous fuzzy sets, fuzzy logic AND, fuzzy rules with linear rule consequent and the generalized defuzzifier. We first prove that the TS fuzzy systems can uniformly approximate any multivariate polynomial arbitrarily well, and then prove they can also uniformly approximate any multivariate continuous function arbitrarily well. We have derived a formula for computing the minimal upper bounds on the number of fuzzy sets and fuzzy rules necessary to achieve the prespecified approximation accuracy for any given bivariate function. A numerical example is furnished. Our results provide a solid-theoretical basis for fuzzy system applications, particularly as fuzzy controllers and models.
---
paper_title: Approximation theory and feedforward networks

paper_content:

Approximation of real functions by feedforward networks of the usual kind is shown to be based on the fundamental principle of approximation by piecewise-constant functions. This principle underlies a simple construction given for three-layer networks and suggests possible difficulties in determining two-layer networks.
---
paper_title: General Takagi-Sugeno fuzzy systems with simplified linear rule consequent are universal controllers, models and filters

paper_content:

Abstract Takagi-Sugeno (TS) fuzzy systems have successfully been employed, mainly in a trial-and-error manner, to solve many control and modeling problems; but their applications as signal filters remain to be fully explored. Compared to their nonfuzzy counterparts, TS fuzzy controllers and models are difficult to be efficiently constructed because there is a large number of design parameters in the rule consequent. The number grows dramatically with the increase of the number of input fuzzy sets and input variables. Furthermore, there exists little published result on relationship between TS fuzzy controllers/models/filters and their nonfuzzy counterparts. In this paper, we investigate, in relation to some popular nonfuzzy controllers, models and filters, analytical structure of a general class of multi-input single-output (MISO) TS fuzzy systems that use arbitrary fuzzy rules with our recently introduced simplified linear rule consequent. Other components of the fuzzy systems in this study are general: arbitrary continuous input fuzzy sets, any type of fuzzy logic AND and the generalized defuzzifier containing the widely used centroid defuzzifier as a special case. We prove that the general MISO TS fuzzy systems are: (1) nonlinear variable gain controllers when implemented as controllers, or (2) nonlinear time-varying auto-regressive with the extra input (ARX) models when implemented as models, or (3) nonlinear infinite impulse response (IIR) or finite impulse response (FIR) filters when implemented as filters. Furthermore, we constructively prove that the general TS fuzzy systems with the simplified linear rule consequent are universal approximators and can approximate any continuous function in closed domain arbitrarily well. The practical implication of these results is that these fuzzy systems, with much less design parameters, are always able to produce solutions to various control, modeling and filtering problems. We also establish sufficient conditions that can be used to calculate the number of input fuzzy sets and rules needed for achieving prespecified approximation accuracy.
---
paper_title: Approximation by superpositions of sigmoidal functions

paper_content:

In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set of affine functionals can uniformly approximate any continuous function ofn real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single hidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.
---
paper_title: A comparative study on sufficient conditions for Takagi-Sugeno fuzzy systems as universal approximators

paper_content:

Universal approximation is the basis of theoretical research and practical applications of fuzzy systems. Studies on the universal approximation capability of fuzzy systems have achieved great progress in recent years. In this paper, linear Takagi-Sugeno (TS) fuzzy systems that use linear functions of input variables as rule consequent and their special case, named simplified fuzzy systems that use fuzzy singletons as rule consequent, are investigated. On condition that overlapped fuzzy sets are employed, new sufficient conditions for simplified fuzzy systems and linear TS fuzzy systems as universal approximators are given, respectively. Then, a comparative study on existing sufficient conditions is carried out with numeric examples.
---

Title: A survey on universal approximation and its limits in soft computing techniques

Section 1: Introduction
Description 1: Provide background information and the historical context related to universal approximation, particularly focusing on its initial hypothesis and subsequent mathematical proofs.

Section 2: Mathematical Notions
Description 2: Summarize the mathematical concepts and notations that are utilized throughout the paper, ensuring readers have a clear understanding of the fundamental terms.

Section 3: Positive results on universal approximation
Description 3: Discuss the successful instances and demonstrations of universal approximation across various soft computing techniques.

Section 3.1: Universal approximation in neural networks
Description 3.1: Detail the positive results specific to neural networks, including key proofs and theorems that establish their universal approximation properties.

Section 3.2: Universal approximation in fuzzy systems
Description 3.2: Outline the findings and key results that showcase the universal approximation capabilities of fuzzy systems.

Section 4: Negative results on universal approximation
Description 4: Present the limitations and challenges of universal approximation, discussing discontinuity issues and nowhere denseness theorems.

Section 4.1: Discontinuity of the best approximation
Description 4.1: Explain the discontinuity problem in obtaining the best approximation and its practical implications.

Section 4.2: Nowhere denseness theorems
Description 4.2: Discuss the nowhere denseness theorems in the context of fuzzy systems and the implications for practical applications.

Section 5: The rate of approximation
Description 5: Analyze the rate at which the approximation converges to the actual function and discuss factors that influence this rate.

Section 6: Constructive results
Description 6: Provide methods and constructive approaches to achieve universal approximation in both neural networks and fuzzy systems.

Section 6.1: Results for neural networks
Description 6.1: Summarize constructive results and approaches specifically related to neural networks, highlighting methods for determining necessary number of hidden units.

Section 6.2: Results for fuzzy systems
Description 6.2: Detail the constructive results for fuzzy systems, including the methodologies for determining the optimal number of rules.

Section 7: Conclusions
Description 7: Summarize the key findings and insights from the survey, emphasizing the state-of-the-art in universal approximation and its practical implications in soft computing.