While self-organizing principles have motivated much of early learning models, such principles have rarely been included in deep learning architectures. Indeed, from a supervised learning perspective it seems that topographic constraints are rather decremental to optimal performance. Here we study a network model that incorporates self-organizing maps into a supervised network and show how gradient learning results in a form of a self-organizing learning rule. Moreover, we show that such a model is robust in the sense of its application to a variety of  areas, which is believed to be a hallmark of biological learning systems. Machine learning has made significant improvements, specifically with deep neural network models BID13 , BID0 , BID4 . Deep learning was made possible by much faster computer technology such as GPUs, and with algorithmic advancement such as BID17 BID3 BID10 , BID2 . Learning tasks that, due to their complexity or data volume, were impossible to execute a decade ago, now can be run in reasonable time scale. The improvements allow the applications of Deep Learning to many real world problems.Learning good internal representations is a key aspect of deep learning. Indeed, it is interesting to recall that the first breakthrough in deep learning came from an application of unsupervised pretraining with gradient-based fine tuning BID7 . Restricted Boltzmann Machines (RBMs) BID6 and Autoencoders BID1 BID8 are utilized for constructing the hidden layers of early models such as Deep Belief Networks (DBN) and Deep Boltzmann Machine (DBM), , BID16 . While much research of deep learning research focuses on learning efficiency and running performances, there is much less research into the understanding of the formation of internal representation in hierarchical neural networks. Moreover, while self-organizing maps have been and integral part of biologically motivated learning theories since the 1970s BID19 , BID12 the role of such self-organizing mechansims are less understood in modern deep learning theories. Topographical self-organization is often observed in biological neural networks BID11 , BID15 and thus may give new insights in understanding learning and self-organization in artificial neural networks.Here, we propose a network that combines aspects of self-organization into a supervised network model for classification. More specifically, in this study we modify the previously proposed Restricted Radial Basis Function Networks (rRBF) BID5 with a softmax output layer that is trained on a crossentropic cost function. This is more consistent with a probabilistic interpretation of the class membership output function than the previous implementation which allows a more clear derivation of the emergence of the self-organizing learning aspects of this network. We call this modified network the Softmax Restricted Radial Basis Function Networks (S-rRBF). Through this network we argue that it is possible to build a learning model in which unsupervised self-organization and supervised learning are just different aspect of a single learning mechanism. We show that the network achieves compatible performance with other deep network architectures while having the added feature of robustness in the sense that it compares favorable with the best performers in the studied examples, while the best performer changes for different applications. While the results are consistent with BID20 'No free lunch theorem', they also highlight that robustness against variation of applications and not the best performance is an important part in flexible learner, which is thought to be of importance when understanding biological learning systems.We highlight our ideas here with well understood applications examples of moderate complexity. However, the proposed architecture can also be scaled to deeper layers and hence applied to deeper learning problems. The main contribution here is showing algebraically the emergence of the selforganizing structures from supervised gradient learning. We believe that this research opens new insights into the relation between unsupervised and supervised learning. Also, we illustrate on some examples the internal representation in the competitive layer and compare it to a standard selforganizing map (SOM) BID12 and to t-Stochastic Neighborhood Embedding (t-SNE) BID18 ) that represents a deeper transformation of the feature space. In this research we showed that it is possible to build a hierarchical neural network that self-organizes with context-relevant topographical internal representation. More specifically, we showed that topographical self-organization can emerge as an implication of the supervised learning. Thus, the two learning processes of self-organization and supervised learning, which are often considered to be unrelated, are can be viewed as two different aspects of a single learning mechanism. The two learning processes are only distinguished by the layers where they occurs. The internal self-organization in this network is not fully unsupervised. However, the direction of the self-organization process in a hidden neuron is only decided by the relative value of the connection weight leading from the neuron to the output neuron relevant to the true label of the input and thus not dependent on the supervised error.The experiments show that the classification performance of the proposed model is comparable to that of standard supervised networks. While the proposed model does not always outperform existing conventional models, we found that the performance was comparable to the best performer for most of the diverse benchmark applications. Specific machine learning methods often perform well on datasets for which they have been designed, but it is well acknowledged that sufficient performance in a variety of tasks is useful in many applications such as robotics and probably to understand better human abilities. Another advantage of our system is its 2-dimensional internal layer offers auxiliary visual information on its learning representations. The S-rRBFcan can readily expanded into deep networks. As layered networks transfer transform inputs (physical stimuli) into labels (concepts) in a layer by layer manner, the visualization of internal layers in multi-layered S-rRBF can be considered as concept-forming visualization. The visualization can potentially offer new insights for machine learning.
integration of self-organization and supervised learning in a hierarchical neural network The paper discusses learning in a neural network with three layers, where the middle layer is topographically organized and investigates interplay between unsupervised and hierarchical supervised learning in biological context. A supervised variant of Kohonen's self-organizing map (SOM), but where the linear output layer is replaced with squared error by a softmax layer with cross-entropy. Proposes a model using hidden neurons with self-organising activation function, whose outputs feed to classifier with softmax output function. 