We investigate the methods by which a Reservoir Computing Network (RCN) learns concepts such as 'similar' and 'different' between pairs of images using a small training dataset and generalizes these concepts to previously unseen types of data. Specifically, we show that an RCN trained to identify relationships between image-pairs drawn from a subset of digits from the MNIST database or the depth maps of subset of visual scenes from a moving camera generalizes the learned transformations to images of digits unseen during training or depth maps of different visual scenes. We infer, using Principal Component Analysis, that the high dimensional reservoir states generated from an input image pair with a specific transformation converge over time to a unique relationship. Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples. Thus, generalization of learning to unseen images is interpretable in terms of clustering of the reservoir state onto the attractor corresponding to the transformation in reservoir space. We find that RCNs can identify and generalize linear and non-linear transformations, and combinations of transformations, naturally and be a robust and effective image classifier. Additionally, RCNs perform significantly better than state of the art neural network classification techniques such as deep Siamese Neural Networks (SNNs) in generalization tasks both on the MNIST dataset and more complex depth maps of visual scenes from a moving camera. This work helps bridge the gap between explainable machine learning and biological learning through analogies using small datasets, and points to new directions in the investigation of learning processes. Different types of Artificial Neural Networks (ANNs) have been used through time for the task of object recognition and classification. Feed-forward structures, such as convolutional neural networks, deep learning BID17 , stacked auto encoders etc. have been extensively studied and are the state of the art for classification. These architectures are well understood due to their feed-forward and non-dynamic nature.However, biological systems such as the visual cortex are known to have primarily ( 70 %) recurrent connections BID2 with less than 1 % of the connections being feedforward Da Costa and Martin (2009). RCN's (or closely related models) provides explanations of why biological brains can carry out accurate computations with an 'inaccurate' and noisy physical substrate BID11 , especially accurate timing BID16 , of the way in which visual spatio-temporal information is super-imposed and processed in primary visual cortex Danko Nikoli c and Maas (2006); BID1 . In addition, biological systems learn visual concepts through analogies, using only a handful of examples BID20 . In particular, in BID9 , bees were trained to fly towards the image from a pair of images that looked very similar to a previously displayed base image. On training bees to fly towards the visually similar image, the bees were presented with two scents, one very similar and one different from a base scent. As a consequence of the visual training that induced preference to the very similar category, the bees flew towards the very similar scent. Thus, biological systems have been found to translate learning of concepts of similarity across sensory inputs, leading us to believe that the brain has a common and fundamental mechanism that comprehends through analogies or through concepts of 'similarity'.Deriving inspiration from nature, we hope to develop a biologically plausible learning technique that learns through analogies.In our framework, we refer to generalization as the ability of a system to learn the relationships or transformations, both linear and non-linear, between a pair of images and be able to recognize the same transformation in unseen image-pairs. Feed-forward networks have, to the best of our knowledge, not been successful in developing an explainable model for this type of generalization of learning. In addition, learning of stand-alone images without drawing comparisons isn't biologically plausible. Networks that require large datasets and hence increasingly powerful GPUs do not scale well. It seems reasonable to say that humans learn through comparitively few training examples BID8 . For instance, a child would learn the features of a horse and the difference between a horse and a donkey, simply by observing at a handful of examples, contrary to deep learning. While research in learning from very few images, one shot learning BID23 etc. has gained momentum recently, integrating it with generalization of learning is a relatively unexplored area.In the ground-breaking work of BID12 , the success of Recurrent Neural Networks (RNNs) depend on the existence of attractors. In training, the dynamical system of the RNN is left running until it ends up in one of its several attractors. Similarly, in BID14 , a unique conceptor is found for each input pattern in a driven RNN. However, training of RNNs is difficult due to problems like the vanishing gradient. BID3 showed that much slower dynamics can be introduced in the RNN by using a random network of neurons with short term plasticity, thus allowing the system to work with training of only the output weights. Exploiting this property, Echo State Networks (ESN) BID13 and Liquid State Machine (LSM) BID18 , commonly falling under Reservoir Computing (RC) were introduced. RC is appealing because of its dynamical property and easy scalability since the recurrent connections in the network aren't trained. Applications of RC include many real world phenomena such as weather or stock market prediction, self driven cars, speech processing and language interpretation, gait generation and motion control in robots etc. RCNs and RNNs perform very well for generating chaotic dynamics BID15 . Models of spontaneously active neural circuits typically exhibit chaotic dynamics, as in RCNs BID19 . Such chaotic dynamics is found in spiking models of spontaneous activity in cortical circuits BID21 .In this work, we train RCNs on both the MNIST handwritten digit database as proof of concept as well as depth maps of visual scenes from a moving camera, to study generalization of the learned transformations between pairs of images. We classify pairs of images into very similar, rotated, zoomed , blurred or different. The reservoir activity is then studied to reveal the underlying features of the activity that are responsible for classification. We find that the relationships between reservoirstate pairs corresponding to input image pairs converge for image pairs with a common relationship between them. In other words, the reservoir only learns relationships between the images , not features of the individual images themselves. This allows for generalization of the learned relationships to all image pairs, seen and unseen by the reservoir. Additionally we compare its performance for a generalization task to a pair-based deep siamese neural network (SNN) built on the keras implementation and show that the reservoir performs significantly better, both for simpler MNIST images as well as for depth maps . We also show that the reservoir is able to recognize linear combinations of the individuals transformations it has learned. This work can useful in the field of computer vision to identify similar transformations between images, even if they are non-linear as in a moving camera, in a biological plausible and computationally efficient way. In this paper we have used RCNs to solve a class of image classification problems that generalize learning of relationships between images using a rather small training data. While image classification has been studied extensively before, here we present a biologically plausible method that not only generalizes learning, but also allows us to interpret the results analytically through a dynamical systems lens. We see that the differential reservoir states obtained from input image-pairs with a common transformation have principal components that are aligned closer together. From a dynamical systems perspective, this can be interpreted as the existence of attractors in reservoir space, each corresponding to a given image transformation. Thus, by reducing the dimensionality of the reservoir space, the reservoir as a dynamical system allows us to train on a much smaller training dataset, whereas contemporary methods such as deep learning require much larger datasets due partly to the lack of dynamics. This same property also allows the reservoir to generalizes the relationships learned to images it hasn't seen during training.In a reservoir, the image space is mapped onto the reservoir space in a way as to preserve the locality of common transformations in reservoir space. In addition, the reservoir performs significantly better than a deep SNN for the task of generalization. From a computation perspective, the reservoir is fast since only the output weights are being trained and the reservoir is sparsely connected. Further, we argue that our method is biologically plausible primarily due to the learning technique based on learning using concepts of similarity from a small training, and secondly due to the dynamics of the reservoir that have been shown to resemble neural cortex activity. We conclude that although state of the art machine learning techniques such as SNNs work exceedingly well for image classification, they do not work as well for generalization of learning, for whch RCNs outperform them, due to their ability to function as a dynamical system with 'memory'. Thus, we see the strength of our work as lying in not only its ability to generalize to untrained images, but also our ability to explain this in terms of the reservoir dynamics and PCA. This relates to new ideas in explainable Artificial Intelligence, a topic that continues to receive traction. An interesting direction would be to explore different reservoir architectures that model the human brain better. Another interesting direction would be to use RCNs to study videos which are naturally temporal, and and investigate how the reservoir generalizes in the action domain. Finally, although we get a fairly good performance with a sparse reservoir and few training images, we predict that as the image complexity increases, a more sophisticated reservoir would be required to match performance.
Generalization of the relationships learnt between pairs of images using a small training data to previously unseen types of images using an explainable dynamical systems model, Reservoir Computing, and a biologically plausible learning technique based on analogies. Claims results of "combining transformations" in the context of RC by using an echo-state network with standard tanh acctivations with the difference that recurrent weights are not trained Novel method of classifying different distorions of MNIST data The paper uses an echo state network to learn to classify image transformations between pairs of images into one of fives classes.