To appear in Journal of Artificial Neural Networks , 1995 . ? This research is funded in part by grants from Novell Inc. and Word Perfect Corp. . An Efficient Transformation for Implementing Two-layer Feedforward Neural Networks George L. Rudolph Tony R. Martinez Computer Science Department Brigham Young University Provo , Utah 84602 Abstract ? Most Artificial Neural Networks -LRB- ANNs -RRB- have a fixed topology during learning , and often suffer from a number of short-comings as a result . Variations of ANNs that use dynamic topologies have shown ability to overcome many of these problems . This paper introduces Location-Independent Transformations -LRB- LITs -RRB- as a general strategy for implementing distributed feedforward networks that use dynamic topologies -LRB- dynamic ANNs -RRB- efficiently in parallel hardware . A LIT creates a set of location-independent nodes , where each node computes its part of the network output independent of other nodes , using local information . This type of transformation allows efficient support for adding and deleting nodes dynamically during learning . In particular , this paper presents an LIT for dynamic Backpropagation networks with a single hidden layer . The complexity of both learning and execution algorithms is O -LRB- n + p + logm -RRB- for a single pattern , where n?is the number of inputs , p is the number of outputs , and m is the number of hidden nodes in the original network . I. INTRODUCTION Hardware support for ANNs is important for handling large , complex problems in real time . Learning times can exceed tolerable limits for applications of large size and complexity using conventional computing schemes . Furthermore , hardware is becoming cheaper and easier to design . The Location-Independent Transformation -LRB- LIT -RRB- is a general implementation strategy for ANNs that overcomes several weaknesses of current hardware implementation methods . Most ANNs use static topologies?the topology is fixed initially , and remains the same throughout learning . ANNs with static topologies often suffer from the following short-comings : ? sensitivity to user-supplied parameters : learning rate -LRB- s -RRB- , etc. , ? local error minima during learning , ? there is no mechanism for deciding on an effective initial topology -LRB- number of nodes , number of layers , etc. -RRB- . Current research is demonstrating the use of dynamic topologies in overcoming these problems -LSB- 2-5 -RSB- , -LSB- 10-13 -RSB- , -LSB- 16 -RSB- . A dynamic topology is one which allows adding and deleting both nodes and weighted connections during learning . Early ANN hardware implementations are model-specific , and are intended to support static topologies -LSB- 6-7 -RSB- , -LSB- 13 -RSB- . More recent neurocomputer systems have specialized neural hardware , and seek to support more general classes of ANNs -LSB- 8 -RSB- , -LSB- 15 -RSB- , -LSB- 21 -RSB- . Although some neurocomputers could potentially support dynamic topologies more directly in hardware , rather than in software , they currently do not . Of course , general parallel machines , like the Connection Machine -LSB- 9 -RSB- and the CRAY -LSB- 1 -RSB- , can simulate the desired dynamics in software , but these machines are not optimized for neural computation . LIT supports general classes of ANNs and dynamic topologies in an efficient parallel hardware implementation . LIT redesigns the original network into a hierarchical , parallel network of LocationIndependent nodes -LRB- LI-nodes -RRB- . A LI-node contains enough information locally to compute its function independent of any other node in the network . The set of LI-nodes is then embedded in a uniform tree topology . Figure 1 gives a general view of LIT . Execution -LRB- running the network -RRB- and learning are based on efficient broadcast/gather operations in the tree . Because the nodes are location-independent , they can be placed anywhere in the tree , and dynamic topologies can be efficiently and easily supported : Adding nodes is as simple as allocating free nodes in the tree ; deleting nodes is as simple as de-allocating the desired nodes . LITs have been developed for counterpropagation and competitive learning -LSB- 18 -RSB- , and Adaptive Self-Organizing Concurrent Systems ? Adaptive Algorithm 2 -LSB- 12 -RSB- , -LSB- 17 -RSB- . LITs can potentially support a broad set of ANNs , thus allowing one efficient implementation strategy to support dynamic variations of many ANNs . This paper presents the transformation for backpropagation -LRB- BP -RRB- networks with a single hidden layer , though the same basic transformation applies to networks with an arbitrary number of layers -LSB- 19 -RSB- . Many dynamic extensions to BP have been proposed in the literature -LSB- 3-5 -RSB- , -LSB- 10 - 11 -RSB- , -LSB- 14 -RSB- , -LSB- 22 -RSB- . The dynamic BP algorithm presented in this paper borrows ideas from -LSB- 14 -RSB- . However , the reader should not assume that this is the only model supported by LIT?rather it is an illustrative example of the LIT strategy . Section II presents a general overview of LIT ; section III demonstrates the transformation of 2-layer BP models , and gives the ? transformed ? versions of algorithms for the standard BP model -LSB- 20 -RSB- ; section IV presents an overview of the dynamic BP model of -LSB- 14 -RSB- , and extends the learning algorithm of section III accordingly ; section V gives a complexity analysis of the algorithms in section III and IV ; section VI is the conclusion . 1 . Transform the ANN 2 . Embed the LI-nodes in a tree arbitrary number of hidden layers LI-Nodes m m Figure 1 . General LIT Transformation II . THE TRANSFORMED NETWORK A transformation redesigns the network so that each node contains enough information locally to compute its part of the network output , independent of any other node . A network whose nodes have this property is said to be location-independent . The nodes also are locationindependent?regardless of the physical location of any node in the network , the relative order in which they compute results , or the order in which those results are gathered , the individual computations are the same , and the network output is the same . Furthermore , because a node?s information is local , adding or deleting nodes from the network can be done without affecting any other nodes . Thus , location-independence allows efficient support for dynamic topologies . Intuitively , the transformation is applied to the intial topology of the original network , before any learning has taken place ; however , it can be viewed as being applied to some `` snapshot '' of the network , at any time , and the process is still the same . In this paper , the term Control Unit?refers to a mechanism that broadcasts inputs to a network , and gathers results from it . The term original refers to a network before it is transformed , and the term transformed refers to a network after it has been transformed . These terms apply similarly to the nodes as well . The number of layers refers to the number of weight layers . LIT is a two-step process : 1 . Construct a set of location-independent nodes based on the original model . 2 . Embed the nodes in a tree . This process is outlined in figure 2 . Based on an original ANN model -LRB- left -RRB- , a set of LI-nodes is constructed -LRB- middle -RRB- , and the nodes are embedded in a tree -LRB- right -RRB- . LIT is not simply a reorganization of the original network , but rather involves redesigning the basic structure . The 4x5x3 original network in figure 2 is transformed to a network with five 4x3 nodes in a tree . A construction typically also involves reformulating the network equations , in order to more particularly describe the behavior of the transformed network . Although the original and transformed networks compute equivalent functions , correspondences between the two are not always direct , nor obvious . Details of a construction vary across different ANNs . 1 2 3 4 5 2 3 54 1 2 4 1 5 3 1 . Transform the ANN 2 . Embed the LI-nodes in a tree Figure 2 . An Example Location-Independent Transformation A communication topology can be chosen as a matter of speed , cost and reliability . The same topology can be used with many models . A binary tree topology is specified in the second step in figures 1 and 2 . It is a simple , fast , regular , hierarchical topology that naturally supports broadcast and gather operations . It provides a basis for applying the transformation and explaining the associated execution and learning algorithms , without obscuring the details with a complex interconnect . However , a hypercube , or some other more fault-tolerant topology , could also be used . Hierarchical topologies allow adding or deleting nodes with time logarithmic in the number of nodes in the network , while limiting connectivity among nodes . The connectivity of a binary tree does not explode because each node is connected to one parent and zero to two children . The algorithms for the transformed network show explicitly how the required communication can take place . An example illustrates the support for dynamic topologies . When a node is added to -LRB- deleted from -RRB- an original network , both the node and its connections to other nodes are added -LRB- deleted -RRB- . If a hidden node 6 were added to the original network in figure 2 , four connections to the input and three connections to the output nodes would also be added . This is accomplished in the transformed network by allocating a free node , such as the left child of node 3 , and initializing it as a 4x3 node with the corresponding weights . The deletion of a hidden node in the original network , such as node 5 , is accomplished in the transformed network by marking the corresponding node as ? free ? . No other nodes are affected . III . TRANSFORMATION OF A 2-LAYER BP NETWORK It is assumed that the reader is familiar with the standard BP model -LSB- 20 -RSB- , however some necessary definitions are given here . Following that is a discussion of the structure and operation of a transformed node . In this description , variables are introduced , used and defined informally . Formal definitions of variables are given in sections III.C and III.D , which give the execution and learning algorithms , respectively . A. BP Definitions In a BP network , there are typically three groups of nodes?input nodes , output nodes , and hidden nodes -LRB- see figure 3 -RRB- . In the standard BP model , the input nodes act as place-holders only , and do not perform any function on the values they receive -LRB- some BP models may use the nodes for preprocessing such as normalization , but this is not part of the basic model -RRB- . A network may have an arbitrary number of layers of weights , but 2-layer and 3-layer networks -LRB- counting by the number of weight layers -RRB- are the most common , and in this paper , only the LIT for 2-layer networks is presented . The LIT for networks with more than two layers is presented in -LSB- 19 -RSB- . Here , and throughout the rest of this paper , given a node j , all the nodes in the same node layer as j are referred to collectively as the current layer . The nodes in the node layer labeled with k are referred to as the subsequent layer . The nodes in the node layer labeled with i ? are referred to as the previous layer . This terminology is straightforward in execution mode or in the feedforward phase of learning . Note that the names remain the same during the backpropagation phase of learning . From the point of view of a hidden node j , the weights on its inputs are referred to collectively as uj , and the weights on its output as wj . i i i i kkk j Figure 3 . Use of the BP equations for an arbitrary hidden node j. B. Transformation Figure 4 shows the transformation of the 4x5x3 network into a 6-node location-independent network -LRB- from figure 2 -RRB- . There are five nodes in the new network , one node in the transformed network for each node in the original hidden layer , plus one bias node . Figure 5 shows a node in the transformed network -LRB- transformed node -RRB- in more detail?the left side shows a node operating during execution mode and the forward phase of learning , while the right side shows the backpropagation phase of learning . The structure and operation of a transformed node -LRB- except the bias node -RRB- and transformed network is best introduced by informal comparison with an original hidden node , node 2 in figure 4 , for example . The original node 2 has four weights from the input layer , and three weights to the output layer . A transformed BP node therefore stores two vectors -LRB- layers -RRB- of weights?one on inputs -LRB- hereafter uj -RRB- and one on outputs -LRB- hereafter wj -RRB- . The transformed node 2 has four weights in uj and three weights in wj . Each weight in the transformed node , uij or wjk , has the value of the corresponding original weight . Each transformed node also stores a bias value qj , which corresponds to the bias value in the respective original hidden node . For purposes of execution and learning , qj can be treated as a component of uj . The bias node is like the other transformed nodes , with some exceptions . It needs no uj , because its activation is always 1 . The elements of its wj each correspond to one of the biases from the nodes in the original output layer . It has no additional bias values of its own . In all other respects , the bias node behaves like the other transformed nodes . 1 2 3 4 5 2 3 4 1 2 4 1 5 3 1 . Transform the ANN 2 . Embed the LI-nodes in a tree 6 6 5 Figure 4 . Transforming a 2-layer BP Network B. 1 . Execution Mode in the Transformed Network The network executes as follows : The Control Unit sends the inputs to the root node , which passes those values onto each of its children , and they to theirs , and so on . Each node -LRB- except the bias node -RRB- receives the broadcast , multiplies each input by the corresponding weight on the input , and sums the results together with each node?s bias . The result of this summation is the same as the original netj . f -LRB- netj -RRB- , the sigmoid of each netj , is then computed at each node . This corresponds to the scalar Oj for a hidden node j in the original network . In the bias node , f -LRB- netj -RRB- is not computed , but is always 1 . The weights on output are each multiplied by f -LRB- netj -RRB- , to produce an output vector which is the new Oj . O2 , the output vector for the transformed node 2 , corresponds to the weights out of the original node 2 each multiplied by the scalar O2 . The vector summation of the transformed nodes ? output vectors is accomplished by a gather operation . Node 2 receives O4 and O5 from its children , does the vector sum of these with O2 , and sends the result to the root node , for example . The components of the vector Y -LRB- the vector sum of all Oj including the Oj of the bias node -RRB- at the root node correspond to netj in each of the output nodes in the original network . Output Input wj1 dw1 d u j backpropagation phase of learning dw2 dw3 wj2 wjp uj1 uj2 u jn Output x1 x2 Input u1j u2j wj1 wj2 xn wjp Ojp j1 O Oj2 s -LRB- -RRB- net j execution mode & forward phase of learning unj Figure 5 . Two Views of a Node in the Transformed Network . The left shows a node operating in execution mode and/or the forward phase of learning . The right shows a node during the backpropagation phase of learning . At this point Y resides at the root node . Now however , the sigmoid of each component of Y needs to be computed , in order to produce values corresponding to the output values of the original output nodes . This vector of output values is called Z. Z can be computed in parallel fashion in two ways : 1 . by a pipelined computation executed as the root node sends Y to the Control Unit , or 2 . by distributing -LRB- through rebroadcasting -RRB- Y to the nodes in the network , where each node computes zi from a component yi , and then gathering Z . There are two main factors involved in deciding how to compute Z?from Y : computation time and communication time . If computation time is fast relative to communication time , then method 1 is more efficient . If , on the other hand , communication is fast relative to computation , the second method is more efficient . For BP , computation of the sigmoid is very fast compared to the time needed to send values to the Control Unit , or to broadcast back to the network?thus , for the BP model , the method using the pipe is preferable . Using the piped scheme then , Z is computed as the root node sends Y to the Control Unit , at which point the Control Unit has the network output . For execution , this pipe has only one computation?the sigmoid?but the pipe for learning -LRB- below -RRB- has more steps . B. 2 . Learning Mode in the Transformed Network -LRB- Standard BP -RRB- As in the original model , learning iterates over a series of patterns , until convergence is reached . As each pattern is presented for learning , the network proceeds through a forward phase and then a backpropagation phase , as follows : First , present the training pattern to the network as above for execution , up to the point at which Y resides at the root node . This is the forward phase . Second , the backpropagation phase begins . The first step of the backpropagation phase is to compute an error value -LRB- d -RRB- for each of the output values , and from that , to compute psse , the single value representing the pattern sumsquared error for the current pattern . This first step is accomplished by a pipe like the one at the end of execution , but with some additional computations . First , Z is computed from Y as above . Second , Z and T are used to compute the error vector dw , which will be sent back to the nodes . -LRB- Target outputs T are at the Control Unit , and therefore are directly available to the pipe . -RRB- Third , dwk -LRB- each component of dw -RRB- is squared to produce dwk2 -LRB- for calculating sum-squared pattern error -RRB- . Fourth , each dwk2 value is summed into psse at the `` end '' of the pipeline -LRB- psse?is used only to check for convergence during learning . -RRB- Each dwk in dw corresponds to the dk computed by the respective output node in the original network . The second step of the backpropagation phase propagates the error backward -LRB- see the right half of figure 5 -RRB- . The vector dw is broadcast to the nodes . This value is then used to compute the required weight change on each weight -LRB- the other information already exists at the node -RRB- . Also at each node , like node 2 , each wjk is multiplied by the corresponding component of dw , and the results are summed . This new value is used to create a d value , duj , with which to calculate changes for each uij -LRB- except in the bias node , which has no u j -RRB- . Once the appropriate computations are completed , all weights can be updated , and the backward phase ends . The network is ready to move on to the next pattern . There are a variety of potential location-independent sets of nodes that could be constructed . The construction given here results in a node structure that is very similar to the original model . An analysis of the original equations and structures reveals a symmetry both in the information used for computation , and in the behavior during forward and backpropagation phases . The LIT structure reflects that symmetry , as well as achieving the other goals of LIT . C. Execution Algorithm The following equations describe execution mode for the 2-layer LIT BP . They are essentially the original equations , but altered to show the behavior of the LIT model . Definitions q number of weight layers in the original network . q = 2 . n number of nodes in the original input layer , size of uj . m number of nodes in the original hidden layer , number of nodes in the transformed network . p number of nodes in the original output layer , size of wj . a real-valued number , it will be either netj or yk . Oj output vector of new node j. Oj intermediate activation value for new node j , it corresponds to Oj for an original node j. netj real-valued sum of the inputs to node j -LSB- equation -LRB- 4 -RRB- -RSB- . f sigmoid function . X pattern clamped on inputs , has size n. xi ith real-valued component of X. uj vector of weights on inputs in a new node j , has size n. uij real-valued ith component of uj . It corresponds in the original network to the weight from input node i to hidden node j. ojk real-valued kth component of Oj . It corresponds in the original network to the weight from hidden node j to output node k , multiplied by the original Oj . wj vector of weights on output in a new node j , has size p. wjk real-valued kth component of wj . It corresponds in the original network to the weight from hidden node j to output node k. Y vector of intermediate activation values , the sum of all Oj . yk real-valued kth component of Y . It corresponds in the original network to the netj of output node k. Z vector of network output values . zk real-valued -LRB- always between 0 and 1 -RRB- kth component of Z . It corresponds in the original network to the output Ok of output node k. Equations f -LRB- a -RRB- = 1 1 + e - a -LRB- 1 -RRB- Oj = f -LRB- netj -RRB- -LRB- 2 -RRB- netj = x i u ij i = 1 n ? + qj -LRB- 3 -RRB- ojk = Ojwjk -LRB- 4 -RRB- yk = o jk j = 1 m ? -LRB- 5 -RRB- zk = f -LRB- yk -RRB- -LRB- 6 -RRB- The main steps of the execution algorithm -LRB- italics -RRB- , numbered for convenient reference , describe the basic BP algorithm . The sub-steps -LRB- 1.1 , 2.1 , etc. -RRB- specify the details of how the LIT network accomplishes these steps . The algorithm is as follows : -LRB- 1 -RRB- Clamp an input pattern on the input nodes . 1.1 Broadcast input vector X to the nodes . -LRB- 2 -RRB- Each node takes the sigmoid of the sum-of-products of its weights and inputs . 2.1 Each node computes its netj , according to equation -LRB- 4 -RRB- , and then computes Oj according to equation -LRB- 3 -RRB- . Oj for the bias node is 1 . 2.2 Each node 's weight vector wj is multiplied by its -LRB- scalar -RRB- Oj to produce its output vector Oj -LSB- equation -LRB- 5 -RRB- -RSB- . 2.3 Each node computes the vector sum of its Oj with the result from each child , and the sum -LRB- a vector -RRB- is sent to its parent . -LRB- The Control Unit is the parent of the root node . -RRB- . This implements equation -LRB- 6 -RRB- . The result is the vector Y. -LRB- 3 -RRB- Wait until the activation values on the output nodes are valid . 3.1 Y is sent -LRB- through the sigmoid pipe -RRB- to the Control Unit . The result Z is the output of the network . D. Learning Algorithm -LRB- Standard BP -RRB- Below are additional equations that describe the learning mode for the LIT BP . The forward phase of learning is identical to execution mode , and so follows the equations given above . Additional Definitions a weight change momentum constant . dw vector of error values for all wj . It has size p. dwk real-valued jth component of dw . It corresponds in the original network to dk for output node k. duj real-valued error value for uj in node j . It corresponds in the original network to dj for hidden node j. T vector of target values for the current pattern being presented . It has size p. tk target value for zk . It corresponds in the original network to the target value for Ok for output node k. psse pattern sum-squared error for the current pattern being presented . tsse total sum-squared error over all patterns in the current learning epoch . ecrit maximum error tolerance for learning . s number of patterns in the training set -LRB- epoch -RRB- . r simple index , used to avoid confusion with i , j , k , etc. . c current training cycle number . h a real-valued learning constant . Equations f ' -LRB- a -RRB- = ? f -LRB- a -RRB- ? a = f -LRB- a -RRB- -LRB- 1-f -LRB- a -RRB- -RRB- -LRB- 7 -RRB- Dwjk -LRB- c -RRB- = hFjdwk + aDwjk -LRB- c-1 -RRB- Duij -LRB- c -RRB- = hxiduj + aDuij -LRB- c-1 -RRB- -LRB- 8 -RRB- dwk = -LRB- tk-zk -RRB- f ' -LRB- yk -RRB- -LRB- 9 -RRB- duj = -LRB- -LRB- dw k w jk k = 1 p ? -RRB- -RRB- f ' -LRB- Oj -RRB- -LRB- 10 -RRB- tsse = psse r r = 1 s ? -LRB- 11 -RRB- psser = dw k 2 k = 1 p ? -LRB- 12 -RRB- As with the execution algorithm above , the steps of the learning algorithm are numbered for convenient reference . The sub-steps give the details of how LIT accomplishes each step . The learning algorithm is as follows : Until Convergence -LRB- tsse < ecrit at the end of an epoch -RRB- -LRB- 1 -RRB- Present a training pattern to the network 1.1 Forward phase of learning?this is same as execution above , up to -LRB- and including -RRB- step 2.3 . -LRB- 2 -RRB- Calculate the error d of the output units -LRB- T - O -RRB- 2.1 As each yk is sent to the Control Unit , the following occurs : 2.1.1 zk is computed -LSB- equation -LRB- 6 -RRB- -RSB- , and received by the Control Unit . 2.1.2 zk and tk are used to compute dwk -LSB- equation -LRB- 9 -RRB- -RSB- , which is received by the Control Unit . 2.1.3 dwk is used to compute dwk2 . 2.1.4 Each dwk2 is summed with the value of psse so far -LSB- equation -LRB- 12 -RRB- -RSB- . -LRB- 3 -RRB- For each hidden layer Calculatedj using dk from the subsequent layer 3.1 The vector dw is broadcast to the nodes . 3.2 duj is calculated according to equation -LRB- 10 -RRB- . -LRB- 4 -RRB- Update the weights 4.1 Calculate Dwjk and Duij according to equation -LRB- 8 -RRB- . 4.2 Change the weights . -LRB- 5 -RRB- Update tsse according to equation -LRB- 11 -RRB- end Step 5 may actually be computed any time after the end of step 2 ? independent of steps 3 or 4 . Most algorithms assume that psse and tsse can be maintained , but give no explicit mechanism for doing so?LIT provides a mechanism . IV . THE DYNAMIC BP MODEL This section gives an overview the dynamic BP -LRB- hereafter DBP -RRB- model proposed by Odri , et al -LSB- 14 -RSB- , then formally extends the learning algoirthm of section III to support these dynamics . The DBP model of -LSB- 14 -RSB- is of interest because it allows the addition and deletion of both nodes and individual weights dynamically during learning . Some dynamic BP models use pruning as a basic technique , while some both grow and prune structures during learning . Some use global state information , and change the topology only after learning is complete , while some use partial , localized information and attempt to change the topology incrementally as learning proceeds . The algorithm of -LSB- 14 -RSB- can be outlined as follows . Details are discussed after the outline . The steps are numbered to correspond with the algorithm of section III , for convenience . Until Convergence -LRB- tsse < ecrit -RRB- 1 . Present a pattern to the network . 2-5 . Perform standard BP learning , including adjusting the weights . 6 . Perform probabilistic weight deletion . 6.1 Each weight computes a probability of self-deletion in the range 0-1 . 6.2 Each weight generates a random value in the range 0-1 . 6.3 If the probability is greater than the random value , then the weight is deleted . 7 . Perform node deletion on the hidden nodes -LRB- the number of input and output nodes remains fixed -RRB- . 7.1 If uj = 0 , or wj = 0 , then node j self-deletes . 7.2 In the case that uj = 0 , and the bias qj ? 0 , then adjust the biases of the remaining nodes to compensate . 8 . Check for node addition -LRB- by probabilistic node division -RRB- . 8.1 Each node computes a probability of dividing , in the range 0-1 . 8.2 Each weight generates a random value in the range 0-1 . 8.3 If the probability is less than the random value , then the node does not divide ? skip step 9 . 9 . Do node division -LRB- only if at least one node divides -RRB- 9.1 Two descendant nodes are created from the node : The uj for both remain the same as the parent , but wj are mutated in order to guarantee that the nodes are unique . end Conceptually , step 9 could be done at the end of step 8 . However , for algorithmic purposes , it is more convenient to check for the possibility of node division separate from actually performing node division . Three ideas need to be discussed in connection with this algorithm : ? Node Addition and Deletion , ? Weight Addition and Deletion , ? Termination of Topological Dynamics . The following definitions are used in the rest of section IV : m a real-valued weight mutation constant . nij random real value generated by -LRB- for -RRB- each wij in checking for weight deletion -LRB- njk for wjk -RRB- . It has range 0-1 . nj random real value generated by each node j in checking for node division . It has range 0-1 . Pijd probability that wij will self-delete -LRB- Pjkd for wjk -RRB- . Pjg probability that node j will divide . kg number of cycles since last node division -LRB- by any node -RRB- . kd number of cycles since last weight was deleted -LRB- in any node -RRB- . bj -LRB- c -RRB- average cumulative change in cell output error at time c , ranging from 0-1 . A. Node Addition and Deletion Nodes are added through a process of probabilistic node division . As learning proceeds , from some time 0 to time c , various d values are computed by each node . At each node j , these values are used to track and compute bj -LRB- c -RRB- , the average cumulative change in node output error at time c. bj -LRB- c -RRB- is directly proportional to the probability that a node will divide at time c during learning , according to the equation Pjg -LRB- c -RRB- = P0gj -LRB- kg -RRB- bj -LRB- c -RRB- . -LRB- 13 -RRB- P0g is the base probability of division -LRB- user-specified -RRB- . kg is a counter -LRB- same value at every node -RRB- that counts the number of cycles -LRB- each pattern presentation is a cycle -RRB- since the last time any node divided . j -LRB- kg -RRB- is a ? recovery control ? function , in the range 0-1 . It is near zero immediately following a division , and near 1 at some user-specified number of cycles after the last division , increasing asymptotically . When a node divides , two ? descendant ? nodes are created , and the ? parent ? deletes itself from the network . The uj for both remain the same as the parent , but wj are mutated according to the equations wjkd1 = -LRB- 1 + m -RRB- wjkp , and wjkd2 = - mwjkp . -LRB- 14 -RRB- d1 and d2?denote the new weight values in the children , and p denotes the original value in the parent . The bias values are the same as the parent?s . These initial values guarantee that the division does not perturb the network , as the net effect of the descendants is the same as the original parent at the time of division . However , as learning proceeds , the two descendants will diverge toward different ? hyperplanes ? across which the parent was oscillating . This increases the overall stability of the network . Nodes delete themselves in two cases : 1 . When ui is 0 , or 2 . when wj is 0 . In case 1 , the threshold or bias value may not be 0 , even if all of the other weights are 0 . -LSB- 14 -RSB- gives a simple transformation which adjusts the biases of nodes in the subsequent layer to compensate for deleting a node with a non-zero bias q ` k = qk + wjkf -LRB- qj -RRB- . -LRB- 15 -RRB- j indicates the node that is deleted , and k indicates the nodes in the subsequent layer . Even though the criteria for deleting a node is deterministic , the deletion of a node is a probabilistic event . It depends on the deletion of weights , which individually delete in probabilistic fashion -LRB- see below -RRB- . B. Weight Addition and Deletion New weights are added to the network automatically when a new node is created . Just as the output error values and changes can be tracked over time , the contribution of a particular weight to a node?s overall activity can be tracked over time . The authors call this value Fij at some time c the factor of action between node i and node j . This factor is computed using the the previous factor value , the current weight value , and the values of outputs from the previous layer at the current time and at one time step earlier . This factor , and its normalized equivalent -RRB- Fij are then used to compute the probability that a weight will self-delete : Pijd -LRB- c -RRB- = ????? P0dj -LRB- kd -RRB- ??? ?? 1 - ? 1 1-e-g ? -LSB- 1-e-vFij -LRB- c -RRB- -RSB- , if ? -RRB- Fij -LRB- c -RRB- ?? F0 0 , if ? -RRB- Fij -LRB- c -RRB- > ? F0 . -LRB- 16 -RRB- F0 is the ? activity threshold ? for a weight . Any weight whose -RRB- Fij remains above this threshold will never self-delete . Over time , this means that only weights that contribute significantly to network activity remain , whereas all others will be forced to 0 . -LRB- A 0 weight is the equivalent of being deleted . -RRB- P0d is the base probability that a weight will self-delete . kd is the number of cycles since the last weight was deleted , analogous to kg for the nodes . g is a sensitivity parameter , and the authors do not explain the role ofv , though clearly it is analogous to g. C. Termination of Topological Dynamics The division of nodes is linked to the amount of error remaining in the network . As the error decreases over learning , the probability that a node -LRB- and thus any node -RRB- will divide will also decrease -LRB- and can reach zero -RRB- . This effectively halts the addition of new nodes , and by implication , weights as well . As mentioned above , weights are not continually deleted from the network . Sufficiently active weights remain in the network , while others are deleted . Once the addition of new nodes ceases , a point will be reached at which the only remaining weights in the network are all above the activity threshold , and will never be deleted . This also halts the deletion of nodes , for reasons explained in section IV.B . D. Dynamic Extensions to the Learning algorithm of III.D . -LRB- 6 -RRB- Check for self-deletion of weights 6.1 Every weight calculates the probability of self-deletingPijd for uij , or Pjkd for wjk , respectively , according to equation -LRB- 16 -RRB- . 6.2 Each weight generates a random value between 0 and 1 , n ij or njk , respectively . 6.3 Any weight uij for which Pijd ? n ij self-deletes . Any weight wjk , for which Pjkd?njk self-deletes . 6.4 If a node has at least one weight self-delete , it sets a deletion flag to 1 ; otherwise the flag is set to 0 . 6.5 Each leaf node sends its deletion flag value to its parent ; each parent ORs its deletion flag with those received from its children , and sends the result to its parent . 6.6 If the Control Unit receives a 1 , then kd is reset to 0 ; otherwise kd is incremented by 1 . -LRB- 7 -RRB- Check for self-deletion of nodes 7.1 Any node j for which uj = 0 or wj = 0 self-deletes . 7.2 If a node self-deletes because uj = 0 , it sets a deletion flag to 1 ; otherwise the flag is set to 0 . 7.3 Adjustment values to compensate for deleted nodes with non-zero biases are computed according to equation -LRB- 15 -RRB- . -LRB- Only nodes that are currently self-deleting compute non-zero results . -RRB- 7.4 Each leaf node sends its deletion flag and adjustment values to its parent . Each parent ORs its deletion flag with those received from children ; each parent sums its adjustment values with those received from its children ; the resulting values are sent up to its parent . 7.5 If the Control Unit receives a deletion flag of 0 , ignore adjustment values and go to step 8 . 7.6 The Control Unit broadcasts the adjustment values to the nodes . 7.7 Each node adjusts its bias according to the equation -LRB- 15 -RRB- . -LRB- 8 -RRB- Check for node division 8.1 Each node j calculates the probability of division Pjg , according to equation -LRB- 13 -RRB- . 8.2 Each node generates a random value between 0 and 1 , nj . 8.3 Any node j for which Pjg?n j , sets its division flag to 1 ; otherwise the flag is set to 0 . 8.4 Each leaf node sends its division flag to its parent ; the parent ORs its division flag with those received from its children ; the result is sent to its parent . 8.5 If the Control Unit receives a division flag of 0 : ? increment division counterkg by 1 , and ? skip step 9 . -LRB- 9 -RRB- If the Control Unit receives a division flag of 1 -LRB- i.e. , at least one node divides -RRB- : 9.1 Reset the division counter kg to 0 . 9.2 Broadcast permission to divide to the nodes . 9.3 Each node which previously set its division flag generates two descendants , according to equation -LRB- 14 -RRB- . 9.4 The parent node is initialized to match one of the children nodes -LRB- thus deleting the parent -RRB- ; the information for the other child is sent to the nearest free child in the subtree of the parent . Step 9 , node division , is potentially the most expensive of all steps in terms of complexity and performance . However , it is important to realize that node division is a rare event in the operation of the network ; when it does occur , it is highly unlikely to tax the resources of the network?particularly large networks . First , only highly unstable nodes have high probabilities of dividing . Second , as training time increases and the network becomes more stable , it is less likely for any particular node to become unstable , and correspondingly more remote that even a few nodes will simultaneously divide . V. COMPLEXITY AND PERFORMANCE The complexity of the algorithms in section IV can be characterized by pipelined broadcast and gather in a tree of nodes . The width of a broadcast block is assumed to be the size of a single variable . The tree has m +1 nodes , one for each hidden node in the original network , plus a bias node , and the nodes operate in parallel . Assume that each node has n inputs and p outputs . Given these assumptions , analysis gives an overall complexity of O -LRB- n + p + logm -RRB- ? for both execution and learning , for a single pattern . The complexity of the steps of the execution algorithm is summarized , followed by elaboration where necessary . The numbers correspond to the steps from the algorithms in section III . The complexity of the learning algorithm is presented in the same format , including the dynamic extensions . Execution Algorithm 1 . Broadcast O -LRB- n + logm -RRB- 2-2 .2 . Node Computation O -LRB- n -RRB- 2.3-3 . Gather O -LRB- p + logm -RRB- total O -LRB- n + p + logm -RRB- Broadcast or gather for a single variable is clearly O -LRB- logm -RRB- . For n?variables , one might expect a complexity of O -LRB- nlogm -RRB- , but the pipelined tree reduces this to O -LRB- n + logm -RRB- , since different levels of the tree operate on different variables simultaneously . The node computation is an O -LRB- n -RRB- operation for n variables , but can be overlapped with broadcast . The sigmoid pipeline at the end of the gather is O -LRB- p -RRB- , one for each output variable , but this is overlapped with the gather of variables up the tree . Learning Algorithm 1 . Forward Phase O -LRB- n + p + logm -RRB- 2 . Compute dw O -LRB- p -RRB- 3.1 Broadcast dw O -LRB- p + logm -RRB- 3.2 Calculate duj O -LRB- p -RRB- 4 . Update weights O -LRB- n + p -RRB- 5 . Partial sum tsse constant 6.1-6 .4 Weight deletion O -LRB- n + p -RRB- 6.5-6 .6 Gather deletion flags O -LRB- logm -RRB- 7.1-7 .2 Node deletion check O -LRB- n + p -RRB- 7.3 Compute Adjustments O -LRB- p -RRB- 7.4-7 .5 Gather Adjustments O -LRB- p + logm -RRB- 7.6 Broadcast Adjustments O -LRB- p + logm -RRB- 7.7 Adjust biases O -LRB- p -RRB- 8.1-8 .3 Node division check O -LRB- n -RRB- 8.4-8 .5 Gather division flags O -LRB- logm -RRB- 9.1-9 .2 Broadcast divide flag O -LRB- logm -RRB- 9.3-9 .4 Allocate new node -LRB- s -RRB- O -LRB- n + p + logm -RRB- total O -LRB- n + p + logm -RRB- Step 2 is O -LRB- p -RRB- because the is one d value for each output variable . Step 4 is O -LRB- n + p -RRB- assuming that a node can update only one weight at a time . Steps 6.1-6 .4 and 7.1-7 .2 also assume that only one weight can be checked at a time . Step 7.3 is O -LRB- p -RRB- because each node generates one value for each of its weights in wj . Steps 7.4-7 .5 have the same character as the gather for execution , only the data is different . Step 7.7 is O -LRB- p -RRB- because p summations are performed at each node . Steps 8.1-8 .3 is O -LRB- n -RRB- because the cumulative error values from each of the weights in uj is used to compute bj -LRB- c -RRB- , etc. . Steps 9.3-9 .4 are O -LRB- n + p + logm -RRB- assuming that there is a free node in the subtree of the parent to hold each new node?n?input variables and p output variables have to be sent to each new node . Probabilistically speaking , only one node will likely divide at any given time c. However , if more than one node does divide , and free nodes exist but are not guaranteed to be in the respective subtrees , then they could be added sequentially , one by one . In this case , Steps 9.3-9 .4 are O -LRB- d -LRB- n + p -RRB- logm -RRB- , where d is the number of nodes that simultaneously divide . The analysis in this section shows that the complexity of execution and learning for the algorithms of section V is O -LRB- n + p + logm -RRB- . In concluding this section , it is interesting to speculate on the relationship between n , p , and m , and the depth of the network logm . Assume that any set of variables -LRB- input or output -RRB- can be represented as strings of binary digits . For present purposes , a single variable can be a binary digit . If the inputs to a node have length n , and the outputs have length p , then there are 2n possible input ? strings ? , and 2p possible output strings . The maximum number of nodes needed to represent all of these is the cross product , namely 2n 2p , or 2 -LRB- n + p -RRB- . Thus , m ? 2 -LRB- n + p -RRB- , and logm ? n + p , where m is the actual number of nodes used . In practice , for an application , it is hoped that at least m ? 2 -LRB- n + p-1 -RRB- , and most often that m ? 2 -LRB- n + p-1 -RRB- , otherwise no significant generalization is taking place as the network learns . This does not mean that logm ? n + p-1 , only that logm < n + p , because the relationship between m and logm is exponential . The above might seem to suggest that the number of input/output variables dominates the performance . However , there are other factors to consider . Except when adding a new node to the network , the total size of any broadcast data is O -LRB- n -RRB- . -LRB- When adding a node , the broadcast size is O -LRB- n + p -RRB- . -RRB- The total size of any gather data is O -LRB- p -RRB- . Given that n < n + p , and n < n + p , is there a meaningful relationship between n , p , and logm -LRB- or the depth of the network -RRB- ? It is tempting to state that both n and p < logm < n + p , under the assumption that for applications of any size , logm will be near n + p . It is easy to come up with examples , in terms of numbers , which make the left half of that relationship false , i.e. for which n > logm . Often , p < n , so that p < logm < n < n + p . In such cases , the number of input variables will dominate broadcast , while the depth of the network will dominate gather . The previous two paragraphs indicate that there are clear upper bounds on complexity and performance , as presented here . Lower bounds , or bounds on actual performance , however , seem to be problem-specific . VI . CONCLUSION ANNs that use a static topology , i.e. a topology that remains fixed throughout learning , suffer from a number of short-comings . Current research is demonstrating the use of dynamic topologies in overcoming some of these problems . The Location-Independent Transformation -LRB- LIT -RRB- is a general strategy for implementing variations of ANNs that use dynamic topologies during learning . This paper introduced ideas for LITs for static and dynamic feedforward , distributed neural networks . An LIT with associated execution and learning algorithms for backpropagation were formally defined , as an example . Analysis gives overall complexities for both execution and learning as O -LRB- n + p + logm -RRB- , where n is the number of inputs , p?is the number of outputs , and m is the number of hidden nodes in the original network . Current work includes the following : ? LITs for other ANNs , aside from those for which transformations have been developed . ? VLSI design and fabrication of LIT models . REFERENCES -LSB- 1 -RSB- G. S. Almasi , A. Gottlieb , Highly Parallel Computing , The Benjamin/Cummings Publishing Company , Inc , Redwood City , CA , 1989 . -LSB- 2 -RSB- P. T. Baffes , J. M. Zelle , ? Growing Layers of Perceptrons : Introducing the Extentron Algorithm ? , Proceedings of 1992 IEEE/INNS International Joint Conference on Neural Networks?Baltimore , Vol . 2 , pp 4979-4984 , 1992 . -LSB- 3 -RSB- T. Denoeux , R. Lengelle , ? Initializing Back Propagation Networks With Prototypes ? Neural Networks , Vol . 6 , # 3 , pp 351-363 , Pergamon Press Ltd , New York , 1993 . -LSB- 4 -RSB- S. Fahlmann , ? Faster-Learning Variations on Back-Propagation : An Empirical Study ? , Proceedings of the 1988 Connectionist Models Summer School , pp 38-51 , 1988 . -LSB- 5 -RSB- S. Fahlmann , C. Lebiere , ? The Cascade-Correlation Learning Architecture ? , Advances in Neural Information Processing 2 , pp 524-532 , Morgan Kaufmann Publishers : Los Altos , CA . -LSB- 6 -RSB- Farhat , N. , D. Psaltis , A. Prata , and E. Paek . Optical Implementation of the Hopfield Model . Applied Optics , Vol . 24 , # 10 . pp.1469-1475 . 1985 . -LSB- 7 -RSB- Graf , H. , L. Jackel , W. Hubbard . VLSI Implementation of a Neural Network Model . In Artificial Neural Networks : Electronic Implementations , Nelson Morgan , Ed . pp. 34-42 . 1990 . -LSB- 8 -RSB- Hammerstrom , D. , W. Henry , M. Kuhn . Neurocomputer System for Neural-Network Applications . In Parallel Digital Implementations of Neural Networks . K. Przytula , V. Prasanna , Eds . Prentice-Hall , Inc. . 1991 . -LSB- 9 -RSB- Hillis , W. Daniel . The Connection Machine . Cambridge , Mass. : MIT Press , 1985 . -LSB- 10 -RSB- E. Karnin , ? A Simple Procedure for Pruning Back-Propagation Trained Neural Networks ? , IEEE Transactions On Neural Networks , Vol . 1 , # 2 , pp 239-242 , June , 1990 . -LSB- 11 -RSB- C. Lee , ? A Simple Procedure for Pruning Back-Propagation Trained Neural Networks ? , Neural Networks , Vol . 6 , # 3 , pp 385-392 . -LSB- 12 -RSB- T. Martinez , D. Campbell , ? A Self-Adjusting Dynamic Logic Module ? , Journal of Parallel and Distributed Computing , Vol . 11 , # 4 , pp 303-313 , 1991 . -LSB- 13 -RSB- C. Mead , Analog VLSI and Neural Systems , Addison-Wesley Publishing Company , Inc , 1991 . -LSB- 14 -RSB- S. Odri , D. Petrovacki , G. Krstonosic , ? Evolutional Development of a Multilevel Neural Network ? , Neural Networks , Vol . 6 , # 4 . pp 583-595 , Pergamon Press Ltd , New York , 1993 . -LSB- 15 -RSB- Ramacher , U. , W. Raab , J. Anlauf , U. Hachmann , J. Beichter , N. Br?ls , M. Wei?ling , E. Schneider , R. M?nner , J. Gl ?? . Multiprocessor and Memory Architecture of the Neurocomputer SYNAPSE-1 . Proceedings , World Congress on Neural Networks 1993 , Vol . 4 . pp. 775-778 . INNS Press , 1993 . -LSB- 16 -RSB- D. Reilly , L. Cooper , C. Erlbaum , ? A Neural Model Category Learning ? , Biological Cybernetics , Vol . 45 , pp 35-41 , 1982 . -LSB- 17 -RSB- G. Rudolph , T. Martinez , ? An Efficient Static Topology for Modeling ASOCS ? , Artificial Neural Networks , Editors , Kohonen et al , pp 279-734 , Elsevier Publishers , North Holland , 1991 . -LSB- 18 -RSB- G. Rudolph , T. Martinez , ? A Transformation for Implementing Localist Neural Networks ? , Submitted , 1994 . -LSB- 19 -RSB- Rudolph G. , Martinez , T. R. ? A Transformation for Implementing Multiple-Hidden-Layer FeedForward Networks ? , Submitted , 1994 . -LSB- 20 -RSB- Rumelhart , D. , J. McClelland , et . al. . Parallel Distributed Processing : Explorations in the Microstructure of Cognition , VOL . 1 , MIT Press , 1986 . -LSB- 21 -RSB- Shams , S. Dream Machine?A Platform for Efficient Implementation of Neural Networks with Arbitrarily Complex Interconnect Structures . Technical Report CENG 92-23 . PhD Dissertation , USC , 1992 . -LSB- 22 -RSB- A. Sperduti , A. Starita . ? Speed Up Learning and Network Optimization With Extended Back Propagation ? , Neural Networks , Vol . 6 , # 3 , pp 365-383 , Pergamon Press Ltd , New York , 1993 .

O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O

backpropagation
dynamic topologies
neural networks

implementation design
reconfigurable architectures