According to parallel distributed processing (PDP) theory in psychology, neural networks (NN) learn distributed rather than interpretable localist representations. This view has been held so strongly that few researchers have analysed single units to determine if this assumption is correct. However, recent results from psychology, neuroscience and computer science have shown the occasional existence of local codes emerging in artificial and biological neural networks. In this paper, we undertake the first systematic survey of when local codes emerge in a feed-forward neural network, using generated input and output data with known qualities. We find that the number of local codes that emerge from a NN follows a well-defined distribution across the number of hidden layer neurons, with a peak determined by the size of input data, number of examples presented and the sparsity of input data. Using a 1-hot output code drastically decreases the number of local codes on the hidden layer. The number of emergent local codes increases with the percentage of dropout applied to the hidden layer, suggesting that the localist encoding may offer a resilience to noisy networks. This data suggests that localist coding can emerge from feed-forward PDP networks and suggests some of the conditions that may lead to interpretable localist representations in the cortex. The findings highlight how local codes should not be dismissed out of hand. Local neural network models, which are often argued to be biologically implausible, have nevertheless been built or discussed by psychologists such as BID11 ; BID16 ; BID13 , and a few researchers like BID1 have done single neuron probing studies (the equivalent of the neuroscience approach) on their neural networks. However, as parallel distributed processing (PDP) neural networks (NN), as discussed by BID17 b) and BID15 , are generally assumed to learn distributed encodings across all situations, it is often believed that a single neuron in an artificial neural network is not interpretable, and experiments to test if this is true are rarely performed.Recently, however, there has been evidence emerging from neuroscience and modern artificial neural networks that demonstrate the existence of interpretable, local codes. BID10 argued that the neurons in the hippocampus codes for information in a highly selective manner in order to learn quickly without forgetting (catastrophic interference), and BID4 argued that some neurons in cortex are highly selective in order to encode multiple items at the same time in shortterm memory (solving the so-called superposition catastrophe). BID14 reported single cells that fire frequently in response to one stimulus, which suggests that individual neurons can be usefully interpreted.Localist codes have been found in artificial neural networks, see BID2 for full reviews, some examples are Le (2013); BID7 ). BID4 have shown that PDP models learn localist codes when trained to co-activate multiple items at the same time. Deep networks learn selective codes under some conditions. For example, BID8 's found quote mark detectors in RNNs. And there is also some evidence from feed-forward models. For example, BID12 have found that probing individual hidden layer neurons (HLNs) with noise and using activation maximisation they can produce a picture of what that neuron will respond most to, and from this they identified HLNs that act as feature detectors, such as those that only responding to creases (in clothing) or eyes or faces and so on.We would like to elucidate the conditions in which simple networks learn selective units, as this may provide further insight into the conditions in which neurons in cortex respond selectively, and, as we expect such codes are learned for sound information theoretic reasons, we expect that these conditions will also apply to when neural networks might learn them as well. Thus, in this paper, we undertake a study of simple feed-forward neural networks to investigate whether local codes, LCs, do actually emerge in PDP networks, and (as we shall show that they do) we then look at what inhibits or promotes the emergence of LCs by designing input and output data with known properties. And, as this data is structured to have some invariance within a class and some randomness, it is proposed that these experiments could as be modelling the layers within the deep neural network above those which transform the input data from pixel space to feature space.To be clear, we consider a neuron to be interpretable if probing of the activation state of it could give correct and useful information about the classification of the input. We look for information about the presence or absence of a category in the hidden layer (category selective HLNs). We separate the qualitative measure (selective) of whether or not a HLN encodes category presence, from the quantitative measure of how much the HLN responds to a category (selectivity). Thus, a HLN is selective if it encodes the presence/absence of a category. Examples are shown in figure 1, as the neuron encodes the presence or absence of the category shown as red circles, equivalently, it could be claimed that these neurons are selective for that category. As biological neurons use energy to encode information, a selective neuron is usually 'selectively on' (see figure 1(left)), but as there is no energy cost in neural networks 'selectively off' units (figure 1(right)) have also been observed. We use the word 'selectivity' as a quantitative measure of the difference between activations for the two categories, A and not-A, where A is the category a neuron is selective for (and not-A being all other categories). Specifically: DISPLAYFORM0 The important point is the qualitative measure of whether or not a neuron is selective, not how much it is selective by, as we are interested in counting the number of local codes that emerge. Note that the chance that all the members of A would emerge disjoint from the members of not-A is 50 50 / 500 50 is tiny (4.32 × 10 −71 ). Furthermore, we found that the selectivity increased with training as the neural network minimised the loss function, but that the number of selective codes did not change once the neural network achieved 100% accuracy.A criticism often made of a grandmother cell hypothesis is that even if a cell fires consistently to a single class, it is not possible to know that it would not have fired to a stimulus that was not presented. For example, although BID14 found a neuron that responded selectively to images of Jennifer Aniston, the authors only presented approximately 100 images to the human participant, and it is possible that other non-tested images would also drive the neuron. BID20 estimated that between 50-150 other images would drive this neuron. Obviously an experimenter cannot present every possible combination of visual inputs to a patient. However, in neural networks with small datasets, we can present all the possible stimuli to the network. We consider a neuron to be selective if it is selective over all the data it is reasonable to expect the network to differentiate between. For example, it is reasonable to do the test over all training data, and it is reasonable to do it over all test and all verification data or even other data of a similar form (such as different photos of the same class), and choosing what constitutes a reasonable set of data is a decision to be made by the experimenters and reviewers. In this work, we chose to use a simple pattern classification task, rather than an image recognition task, as we could then test the NN with all possible patterns. We have demonstrated that interpretable localised encodings of the presence/absence of some categories can emerge from the hidden layer of a feed-forward neural network. As the number of local codes follows a well-defined pattern with the size of the hidden layer, and it is affected by modifications of the input and output data, it suggests that the number of local codes is related to the computing capacity of the neural network and the difficulty of the problem presented to it, suggesting that the local codes offer some modification to the computing power of the neural network.Furthermore, as the hidden layer size increases, there is so much extra capacity that local codes are not needed. Our results suggest that local codes require more effort to train, but offer more efficient use of the available capacity.As the number of local codes shared invariants within a categories, it does imply that the local codes have some function associated with recognising these invariants. As the average number and range of LCs generally increases with dropout, and the LCs are repressed by a fully locally encoded output layer, it suggests that some local codes are good to have, and that number increases with noisy networks. The fact that the dropout data seems to contain multiple overlapping peaks, and, in our tests, peak numbers of LCs are seen at 500 and 1000 (and 2000 for the S R = 1 9 data) HLNs implies that there are more than one qualitative approaches for the neural network to solve the problem, and tuning the problem and neural network parameters nudges the solution to different distributions of local codes. Do these simple networks tell us anything about deep neural networks? The data presented here was designed to have invariant feature 'short-cuts' that the neural network could make use of in classifying input data into classes and the argument could well be made that the data passed between layers of a deep neural network is not of the same quality. Whilst an obvious next experiment for us is to investigate the qualities of the data passed within a neural network, preliminary feed-forward neural network training on standard simple neural network data (such as the Iris dataset from Fisher (1936)) results also show the emergence of local codes when there is a 'short-cut' in the data (publication in preparation). The observations that local codes are seen under dropout, with distributed input and output codes, when there are invariant features and local codes are inhibited with 1-hot output encodings, suggests that local codes might be found in a the middle and higher layers of a deep network, and not the penultimate layer where the 1-hot output could inhibit them or the early layers where invariant features common to a class have not yet been identified. Another interesting question is whether the local codes might have a diagnostic use, for example, is it the case that they increased in networks that generalise or are they, perhaps, an indicator of over-training? Answering this would also highlight when and where we should expect invariants in the data, as learning an invariant feature, such as, 'presence of eyes implies presence of face', could help with generalisation and classification, however learning an irrelevant invariant feature, such as, presence of 'blue sky implies tanks' would not. Discriminating a blue-sky selective neuron from a tank-selective neuron in such a case would require careful thought about what we should consider reasonable data to test for selectivity.
Local codes have been found in feed-forward neural networks A method for determining to what degree individual neurons in a hidden layer of an MLP encode a localist code, which is studied for different input representations. Studies the development of localist representations in the hidden layers of feed-forward neural networks.