Neural Distributed Autoassociative Memories: A Survey
---
paper_title: Informational characteristics of neural networks capable of associative learning based on Hebbian plasticity

paper_content:

The informational aspects of neural networks performing associative memory following Hebbian learning are analysed in detail. The recall process is decomposed into its recognition and error correction components, and the respective contributions are clarified and computed.The analysis identifies the principal sources of information loss. They are shown to be in the choice of the decoding procedure, in the indeterminacy introduced by zero modification states of the synapses and in the statistical dependences between different synaptic modification states.A wide range of storage schemes and decoding procedures are discussed and their optimal characteristics are compared and evaluated relative to the corresponding limits prescribed by Shannon's theorem.
---
paper_title: Content-Addressable Memories

paper_content:

1 Associative Memory, Content Addressing, and Associative Recall.- 1.1 Introduction.- 1.1.1 Various Motives for the Development of Content-Addressable Memories.- 1.1.2 Definitions and Explanations of Some Basic Concepts.- 1.2 The Two Basic Implementations of Content Addressing.- 1.2.1 Software Implementation: Hash Coding.- 1.2.2 Hardware Implementation: The CAM.- 1.3 Associations.- 1.3.1 Representation and Retrieval of Associated Items.- 1.3.2 Structures of Associations.- 1.4 Associative Recall: Extensions of Concepts.- 1.4.1 The Classical Laws of Association.- 1.4.2 Similarity Measures.- 1.4.3 The Problem of Infinite Memory.- 1.4.4 Distributed Memory and optimal Associative Mappings.- 1.4.5 Sequential Recollections.- 2 Content Addressing by Software.- 2.1 Hash Coding and Formatted Data Structures.- 2.2 Hashing Functions.- 2.3 Handling of Collisions.- 2.3.1 Some Basic Concepts.- 2.3.2 Open Addressing.- 2.3.3 Chaining (Coalesced).- 2.3.4 Chaining Through a Separate overflow Area.- 2.3.5 Rehashing.- 2.3.6 Shortcut Methods for Speedup of Searching.- 2.4 Organizational Features and Formats of Hash Tables.- 2.4.1 Direct and Indirect Addressing.- 2.4.2 Basic Formats of Hash Tables.- 2.4.3 An Example of Special Hash Table Organization.- 2.5 Evaluation of Different Schemes in Hash Coding.- 2.5.1 Average Length of Search with Different Collision Handling Methods.- 2.5.2 Effect of Hashing Function on the Length of Search.- 2.5.3 Special Considerations for the Case in Which the Search Is Unsuccessful.- 2.6 Multi-Key Search.- 2.6.1 Lists and List Structures.- 2.6.2 An Example of Implementation of Multi-Key Search by Hash Index Tables.- 2.6.3 The Use of Compound Keywords in Hashing.- 2.7 Implementation of Proximity Search by Hash Coding.- 2.8 The TRIE Memory.- 2.9 Survey of Literature on Hash Coding and Related Topics.- 3 Logic Principles of Content-Addressable Memories.- 3.1 Present-Day Needs for Hardware CAMs.- 3.2 The Logic of Comparison Operations.- 3.3 The All-Parallel CAM.- 3.3.1 Circuit Logic of a CAM Bit Cell.- 3.3.2 Handling of Responses from the CAM Array.- 3.3.3 The Complete CAM Organization.- 3.3.4 Magnitude Search with the All-Parallel CAM.- 3.4 The Word-Parallel, Bit-Serial CAM.- 3.4.1 Implementation of the CAM by the Linear-Select Memory Principle.- 3.4.2 Skew Addressing.- 3.4.3 Shift Register Implementation.- 3.4.4 The Results Storage.- 3.4.5 Searching on More Complex Specifications.- 3.5 The Word-Serial, Bit-Parallel CAM.- 3.6 Byte-Serial Content-Addressable Search.- 3.6.1 Coding by the Characters.- 3.6.2 Specifications Used in Document Retrieval.- 3.6.3 A Record-Parallel, Byte-Serial CAM for Document Retrieval.- 3.7 Functional Memories.- 3.7.1 The Logic of the Bit Cell in the FM.- 3.7.2 Functional Memory 1.- 3.7.3 Functional Memory 2.- 3.7.4 Read-only Functional Memory.- 3.8 A Formalism for the Description of Micro-Operations in the CAM.- 3.9 Survey of Literature on CAMs.- 4 CAM Hardware.- 4.1 The State-of-the-Art of the Electronic CAM Devices.- 4.2 Circuits for All-Parallel CAMs.- 4.2.1 Active Electronic Circuits for CAM Bit Cells.- 4.2.2 Cryotron-Element CAMs.- 4.2.3 Josephson Junctions and SQUIDs for Memories.- 4.3 Circuits for Bit-Serial and Word-Serial CAMs.- 4.3.1 Semiconductor RAM Modules for the CAM.- 4.3.2 Magnetic Memory Implementations of the CAM.- 4.3.3 Shift Registers for Content-Addressable Memory.- 4.3.4 The Charge-Coupled Device (CCD).- 4.3.5 The Magnetic-Bubble Memory (MBM).- 4.4 Optical Content-Addressable Memories.- 4.4.1 Magneto-Optical Memories.- 4.4.2 Holographic Content-Addressable Memories.- 5 The CAM as a System Part.- 5.1 The CAM in Virtual Memory Systems.- 5.1.1 The Memory Hierarchy.- 5.1.2 The Concept of Virtual Memory and the Cache.- 5.1.3 Memory Mappings for the Cache.- 5.1.4 Replacement Algorithms.- 5.1.5 Updating of Multilevel Memories.- 5.1.6 Automatic Control of the Cache Operations.- 5.1.7 Buffering in a Multiprocessor System.- 5.1.8 Additional Literature on Memory Organizations and Their Evaluation.- 5.2 Utilization of the CAM in Dynamic Memory Allocation.- 5.2.1 Memory Map and Address Conversion.- 5.2.2 Loading of a Program Segment.- 5.3 Content-Addressable Buffer.- 5.4 Programmable Logic.- 5.4.1 RAM Implementation.- 5.4.2 CAM Implementation.- 5.4.3 FM Implementation.- 5.4.4 Other Implementations of Programmable Logic.- 5.4.5 Applications of the CAM in Various Control Operations.- 6 Content-Addressable Processors.- 6.1. Some Trends in Content-Addressable Memory Functions.- 6.2 Distributed-Logic Memories (DLMs).- 6.3 The Augmented Content-Addressable Memory (ACAM).- 6.4 The Association-Storing Processor (ASP).- 6.5 Content-Addressable Processors with High-Level Processing Elements.- 6.5.1 The Basic Array Processor Architecture.- 6.5.2 The Associative Control Switch (ACS) Architecture.- 6.5.3 An Example of Control-Addressable Array Processors: RADCAP.- 6.5.4 An Example of Content-Addressable Ensemble Processors: PEPE.- 6.6 Bit-Slice Content-Addressable Processors.- 6.6.1 The STARAN Computer.- 6.6.2 Orthogonal Computers.- 6.7 An Overview of Parallel Processors.- 6.7.1 Categorizations of Computer Architectures.- 6.7.2 Survey of Additional Literature on Content-Addressable and Parallel Processing.- 7 Review of Research Since 1979.- 7.1 Research on Hash Coding.- 7.1.1 Review Articles.- 7.1.2 Hashing Functions.- 7.1.3 Handling of Collisions.- 7.1.4 Hash Table Organization.- 7.1.5 Linear Hashing.- 7.1.6 Dynamic, Extendible, and External Hashing.- 7.1.7 Multiple-Key and Partial-Match Hashing.- 7.1.8 Hash-Coding Applications.- 7.1.9 Hash-Coding Hardware.- 7.2 CAM Hardware.- 7.2.1 CAM Cells.- 7.2.2 CAM Arrays.- 7.2.3 Dynamic Memories.- 7.2.4 CAM Systems.- 7.3 CAM Applications.- 7.4 Content-Addressable Parallel Processors.- 7.4.1 Architectures for Content-Addressable Processors.- 7.4.2 Data Base Machines.- 7.4.3 Applications of Content-Addressable Processors.- 7.5 Optical Associative Memories.- 7.5.1 General.- 7.5.2 Holographic Content-Addressable Memories.- References.
---
paper_title: Storage capacity of generalized networks

paper_content:

We discuss the thermodynamic phases and storage capacity of models which are extensions of the Hopfield-Little model of associative memory, generalized to allow for arbitrary polynomial Hamiltonians. The storage capacity increases as an exponential function of the highest power appearing in the polynomial Hamiltonian. We include the effects of replica symmetry breaking in our computations.
---
paper_title: ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH THE HEBBIAN LEARNING RULE

paper_content:

We consider the Hopfield model with the most simple form of the Hebbian learning rule, when only simultaneous activity of pre- and post-synaptic neurons leads to modification of synapse. An extra inhibition proportional to full network activity is needed. Both symmetric nondiluted and asymmetric diluted networks are considered. The model performs well at extremely low level of activity p<K−1/2, where K is the mean number of synapses per neuron.
---
paper_title: Associative memories with "killed" neurons: The methods of recovery

paper_content:

We consider re-learning ability of a Hopfield-type network after killing some neurons. Neurons were "killed" by means of nullification of corresponding rows and columns of the synaptic matrix. We show that one can restore recognition ability of this network using re-training with the vectors, which were memorized before. The number of vectors needed is equal to the number of deleted neurons. It does not depend on network's size and on volume of stored data.
---
paper_title: Informational characteristics of neural networks capable of associative learning based on Hebbian plasticity

paper_content:

The informational aspects of neural networks performing associative memory following Hebbian learning are analysed in detail. The recall process is decomposed into its recognition and error correction components, and the respective contributions are clarified and computed.The analysis identifies the principal sources of information loss. They are shown to be in the choice of the decoding procedure, in the indeterminacy introduced by zero modification states of the synapses and in the statistical dependences between different synaptic modification states.A wide range of storage schemes and decoding procedures are discussed and their optimal characteristics are compared and evaluated relative to the corresponding limits prescribed by Shannon's theorem.
---
paper_title: Time of searching for similar binary vectors in associative memory

paper_content:

Times of searching for similar binary vectors in neural-net and traditional associative memories are investigated and compared. The neural-net approach is demonstrated to surpass the traditional ones even if it is implemented on a serial computer when the entropy of a space of signals is of order of several hundreds and the number of stored vectors is vastly larger than the entropy.
---
paper_title: Nondirect convergence radius and number of iterations of the Hopfield associative memory

paper_content:

Considers a Hopfield associative memory consisting of n neurons, designed to store an m-set of n-dimensional /spl plusmn/1 statistically independent uniformly distributed random vectors (fundamental memories), using a connection matrix, constructed by the usual Hebbian rule. Previous results have indicated that the maximal value of m, such that almost all m vectors are stable points of the memory, in probability (i.e., with probability approaching one as n approaches infinity), is n/(2 log n)(n/(4 log n) if all m vectors must be stable simultaneously, in probability). Previous work further analyzed the direct convergence (i.e., convergence in one iteration) error-correcting power of the Hopfield memory. The present authors rigorously analyze the general case of nondirect convergence, and prove that in the m=n/(2 log n) case, independently of the operation mode used (synchronous or asynchronous), almost all memories have an attraction radius of size n/2 around them (in the n/(4 log n) case, all memories have such an attraction radius, in probability). This result, which was conjectured in the past but was never proved rigorously, combined with an old converse result that the network cannot store more than n/(2 log n)(n/(4 log n)) fundamental memories, gives a full picture of the error-correcting power of the Hebbian Hopfield network. The authors also upper bound the number of iterations required to achieve convergence. >
---
paper_title: The capacity of the Hopfield associative memory

paper_content:

Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory. Such a memory stores n -tuple of \pm 1 's. The components change depending on a hard-limited version of linear functions of all other components. With symmetric connections between components, a stable state is ultimately reached. By building up the connection matrix as a sum-of-outer products of m fundamental memories, one hopes to be able to recover a certain one of the m memories by using an initial n -tuple probe vector less than a Hamming distance n/2 away from the fundamental memory. If m fundamental memories are chosen at random, the maximum asympotic value of m in order that most of the m original memories are exactly recoverable is n/(2 \log n) . With the added restriction that every one of the m fundamental memories be recoverable exactly, m can be no more than n/(4 \log n) asymptotically as n approaches infinity. Extensions are also considered, in particular to capacity under quantization of the outer-product connection matrix. This quantized memory capacity problem is closely related to the capacity of the quantized Gaussian channel.
---
paper_title: Statistical Mechanics of Neural Networks near Saturation

paper_content:

The Hopfield model of a neural network is studied near its saturation, i.e., when the number p of stored patterns increases with the size of the network N, as p = αN. The mean-field theory for this system is described in detail. The system possesses, at low α, both a spin-glass phase and 2p dynamically stable degenerate ferromagnetic phases. The latter have essentially full macroscopic overlaps with the memorized patterns, and provide effective associative memory, despite the spin-glass features. The network can retrieve patterns, at T = 0, with an error of less than 1.5% for α <αc = 0.14. At αc the ferromagnetic (FM) retrieval states disappear discontinuously. Numerical simulations show that even above αc the overlaps with the sored patterns are not zero, but the level of error precludes meaningful retrieval. The difference between the statistical mechanics and the simulations is discussed. As α decreases below 0.05 the FM retrieval states become ground states of the system, and for α < 0.03 mixture states appear. The level of storage creates noise, akin to temperature at finite p. Replica symmetry breaking is found to be salient in the spin-glass state, but in the retrieval states it appears at extremely low temperatures, and is argued to have a very weak effect. This is corroborated by simulations. The study is extended to survey the phase diagram of the system in the presence of stochastic synaptic noise (temperature), and the effect of external fields (neuronal thresholds) coupled to groups of patterns. It is found that a field coupled to many patterns has a very limited utility in enhancing their learning. Finally, we discuss the robustness of the network to the relaxation of various underlying assumptions, as well as some new trends in the study of neural networks.
---
paper_title: Neural networks and physical systems with emergent collective computational abilities

paper_content:

Abstract ::: Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.
---
paper_title: Associative memory in asymmetric diluted network with low level of activity

paper_content:

We extend the analysis of asymmetric diluted networks to the case of low-activity level. The same learning algorithm which was used for the symmetric model turns out to be successful. The use of "V-variables" (V = 0; 1) leads to significant enhancing of the storage capacity. The overloading phase transition is found to be of the first order, which means good retrieval quality in all associative memory phases. The intensity of time-dependent nonthermal noise can be diminished considerably by the appropriate choice of the neural threshold. Some sort of "universality" of the performance of the networks with low-activity level can be noted.
---
paper_title: Transients and basins of attraction in neutral network models

paper_content:

Approximate dynamic mean field equations for a generalized Hopfield model are derived, which allow to calculate transient properties of this model. These equations are exact for short times and yield the replica symmetric solution as a stationary solution. They allow reliable computation of retrieval trajectories and basins of attraction of retrieval states, as demonstrated by comparison with simulations. The equations are derived for networks with arbitrary mean activity and results are given for the standard model and for low activities.
---
paper_title: Informational capacity and recall quality in sparsely encoded Hopfield-like neural network: analytical approaches and computer simulation

paper_content:

Abstract A sparsely encoded Hopfield-like attractor neural network is investigated analytically and by computer simulation. Informational capacity and recall quality are evaluated. Three analytical approaches are used: replica method (RM); method of statistical neurodynamics (SN); and single-step approximation (SS). Computer simulation confirmed the good accuracy of RM and SN for all levels of network activity. SS is accurate only for large sparseness. It is shown that informational capacity monotonically increases when sparseness increases, while recall quality changes nonmonotonically: initially it decreases and then increases. Computer simulation revealed the main features of network behaviour near the saturation which are not predicted by the used analytical approaches. © 1997 Elsevier Science Ltd.
---
paper_title: On the capacity of an associative memory model based on neural cliques

paper_content:

Based on recent work by Gripon and Berrou (2011), we introduce a new model of an associative memory. We give upper and lower bounds on the memory capacity of the model.
---
paper_title: The storage capacity of the Hopfield model and moderate deviations

paper_content:

This note relates the storage capacity of the Hopfield model of neural networks with possibly correlated patterns to a moderate deviation upper bound for the empirical correlation of the patterns. Examples are, among others, independent patterns with spins that are correlated as in an Ising model or as in a Curie-Weiss model.
---
paper_title: Iterative retrieval of sparsely coded associative memory patterns

paper_content:

Abstract We investigate the pattern completion performance of neural auto-associative memories composed of binary threshold neurons for sparsely coded binary memory patterns. By focussing on iterative retrieval, we are able to introduce effective threshold control strategies. These are investigated by means of computer simulation experiments and analytical treatment. To evaluate the systems performance we consider the completion capacity C and the mean retrieval errors. The asymptotic completion capacity values for the recall of sparsely coded binary patterns in one-step retrieval is known to be ln 2 4 ≈ 17.32% for binary Hebbian learning, and 1 (8 ln 2) ≈ 18% for additive Hebbian learning. These values are accomplished with vanishing error probability and yet are higher than those obtained in other known neural memory models. Recent investigations on binary Hebbian learning have proved that iterative retrieval as a more refined retrieval method does not improve the asymptotic completion capacity of one step retrieval. In a finite size auto-associative memory we show that iterative retrieval achieves higher capacity and better error correction than one-step retrieval. One-step retrieval produces high retrieval errors at optimal memory load. Iterative retrieval reduces the retrieval errors within a few iteration steps (t ⩽ 5). Experiments with additive Hebbian learning show that in the finite model, binary Hebbian learning exhibits much better performance. Thus the main concern of this paper is binary Hebbian learning. We examine iterative retrieval in experiments with up to n = 20,000 threshold neurons. With this system size one-step retrieval yields a completion capacity of about 16%, the second retrieval step increases this value to 17.9% and with iterative retrieval we obtain 19%. The first two retrieval steps in the finite system have also been treated analytically. For one-step retrieval the asymptotic capacity value is approximated from below with growing system size. In the second retrieval step (and as the experiments suggest also for iterative retrieval) the finite size behaviour is different. The capacity exceeds the asymptotic value, reaches an optimum for finite system size, and decreases to the asymptotic limit.
---
paper_title: Advantages of dilution in the connectivity of attractor networks in the brain

paper_content:

Abstract A fundamental question about brain function is why the connectivity in the cortex is diluted, in that neurons in a local region of the neocortex and in the CA3 part of the hippocampal cortex typically have a probability of having a synaptic connection between them that is less than 0.1. In both these types of cortex, there is evidence that the excitatory interconnections between neurons are associatively modifiable, and that the system supports attractor dynamics that enable memories to be stored, which are used in for example short-term memory and in episodic memory. The hypothesis proposed is that the diluted connectivity allows biological processes that set up synaptic connections between neurons to arrange for there to be only very rarely more than one synaptic connection between any pair of neurons. If probabilistically there were more than one connection between any two neurons, it is shown by simulation of an autoassociation attractor network that such connections would dominate the attractor states into which the network could enter and be stable, thus strongly reducing the memory capacity of the network (the number of memories that can be stored and correctly retrieved), below the normal large capacity for diluted connectivity. Diluted connectivity between neurons in the cortex thus has an important role in allowing high capacity of memory networks in the cortex, and helping to ensuring that the critical capacity is not reached at which overloading occurs leading to an impairment in the ability to retrieve any memories from the network. This intra-area diluted connectivity complements the diluted connectivity in the feedforward connections between cortical areas that helps the representations built by competitive learning to be stable.
---
paper_title: Learning and pattern recognition in spin glass models

paper_content:

Spin glass models have a complex phase space which may be used to store information. By an asynchronous relaxational dynamics noisy patterns are recognized very fast. In particular the Hopfield model which may be a simple model for neural networks is analyzed in detail. By numerical simulations and analytical approximations the number of patterns to be stored and the amount of noise to be recognized is calculated. It is demonstrated that a random network can learn patterns by reducing frustrated bonds.
---
paper_title: Characteristics of sparsely encoded associative memory

paper_content:

Abstract Characteristics of an autocorrelation or crosscorrelation associative memory largely depend on how items are encoded in pattern vectors to be stored. When most of the components of encoded patterns to be stored are 0 and only a small ratio of the components are 1, the encoding scheme is said to be sparse. The memory capacity and information capacity of a sparsely encoded associative memory are analyzed in detail, and are proved to be in proportion of n 2 log n 2 , n being the number of neurons, which is very large compared to the ordinary non-sparse encoding scheme of about 0.15n. Moreover, it is proved that the sparsely encoded associative memory has a large basin of attraction around each memorized pattern, when and only when an activity control mechanism is attached to it.
---
paper_title: Sharp upper bounds on perfect retrieval in the Hopfield model

paper_content:

We prove a sharp upper bound on the number of patterns that can be stored in the Hopfield model if the stored patterns are required to be fixed points of the gradient dynamics. We also show corresponding bounds on the one-step convergence of the sequential gradient dynamics. The bounds coincide with the known lower bounds and confirm the heuristic expectations. The proof is based on a crucial idea of Loukianova (1997) using the negative association properties of some random variables arising in the analysis.
---
paper_title: On setting unit thresholds in an incompletely connected associative net

paper_content:

The associative net is a mahix model of associative memory whichhas ah efficiency of recall approaching that of a random access memory with no associative capability. This is a fully connected netwoIk, which m&es it possible to use a simple strategy for setting ,the thresholds of the units in recall. However, most brain structures that a!e thought to underlie leaming and memory have only partial interconnectivity. We deicribe five different strategies for setting the thresholds of units in piilially connected nets. The simprest is a, meZhaniSm of the winners-lake-all type. The most sophisticated strategy employs information about the density of modified synapses and their dishiburion on each output unit. The action Of this mechanism is shown to be equivalent to minimiration of output ermr but without re&ring numerical solution of a selof equations, which would 6e biolo&cally implausible. Simulation results demonstrare the superiority of this mechanism in a typical case. Paramew sensitivity analysis of this mechanism shows~its robusuless and that performance improves as mnnectivity becomes more complete.
---
paper_title: Associative memory: on the (puzzling) sparse coding limit

paper_content:

Recent studies of the information capacity in a sparsely coded memory net has led to some contradictory results. In the Willshaw model, where the couplings are binary (0 or 1), the maximal quantity of information that can be stored is 1n 2 approximately=0.69 bits per synapse. On the other hand a calculation a la Gardner (1988) for (0,1) couplings gives an upper bound for the maximal capacity of about 0.29 bits per synapse. In this study, the author considers two possible sources for this discrepancy. The first one is that the criteria for defining the maximal capacity are different (with or without a constraint of perfect errorless storage). The second one is a difference in the choice of the probability distribution of the random patterns used to compute this capacity. This analysis shows in particular that for the Willshaw model the maximal information capacity is much larger when the number of active neurons is exactly the same in every stored pattern, than when it is given only in average. In addition he gives an argument showing that this result may be generic, e.g., valid for any activity level and independent of the learning rule.
---
paper_title: A Comparative Study of Sparse Associative Memories

paper_content:

We study various models of associative memories with sparse information, i.e. a pattern to be stored is a random string of 0s and 1s with about \(\log N\) 1s, only. We compare different synaptic weights, architectures and retrieval mechanisms to shed light on the influence of the various parameters on the storage capacity.
---
paper_title: Informational characteristics of neural networks capable of associative learning based on Hebbian plasticity

paper_content:

The informational aspects of neural networks performing associative memory following Hebbian learning are analysed in detail. The recall process is decomposed into its recognition and error correction components, and the respective contributions are clarified and computed.The analysis identifies the principal sources of information loss. They are shown to be in the choice of the decoding procedure, in the indeterminacy introduced by zero modification states of the synapses and in the statistical dependences between different synaptic modification states.A wide range of storage schemes and decoding procedures are discussed and their optimal characteristics are compared and evaluated relative to the corresponding limits prescribed by Shannon's theorem.
---
paper_title: ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES

paper_content:

The simple learning algorithm in the neural network with binary synapses, which take one step for storing one pattern is considered. The resulting model turns out to be palimpsestic, and the number of patterns which can be effectively retrieved is L~N1/2.
---
paper_title: Modeling Brain Function: The World of Attractor Neural Networks

paper_content:

From the Publisher: ::: Substantial progress in understanding memory, the learning process, and self-organization by studying the properties of models of neural networks have resulted in discoveries of important parallels between the properties of statistical, nonlinear cooperative systems in physics and neural networks.
---
paper_title: On associative memory

paper_content:

The information storing capacity of certain associative and auto-associative memories is calculated. For example, in a 100×100 matrix of 1 bit storage elements more than 6,500 bits can be stored associatively, and more than 688,000 bits in a 1,000×1,000 matrix. Asymptotically, the storage capacity of an associative memory increases proportionally to the number of storage elements. The usefulness of associative memories, as opposed to conventional listing memories, is discussed — especially in connection with brain modelling.
---
paper_title: Willshaw model: Associative memory with sparse coding and low firing rates.

paper_content:

The Willshaw model of associative memory, implemented in a fully connected network with stochastic asynchronous dynamics, is studied. In addition to Willshaw's learning rule, the network contains uniform synaptic inhibition, of relative strength K, and negative neural threshold -\ensuremath{\theta},\ensuremath{\theta}g0. The P stored memories are sparsely coded. The total number of on bits in each memory is Nf, where f is much smaller than 1 but much larger than lnN/N. Mean-field theory of the system is solved in the limit where C==exp(-${f}^{2}$P) is finite. Memory states are stable (at zero temperature), as long as Cg${h}_{0}$==K-1+\ensuremath{\theta} and ${h}_{0}$g0. When C${h}_{0}$ or ${h}_{0}$0, P retrieval phases, highly correlated with the memory states, exist. These phases are only partially frozen at low temperature, so that the full memories can be retrieved from them by averaging over the dynamic fluctuations of the neural activity. In particular, when ${h}_{0}$0 the retrieval phases at low temperatures correspond to freezing of most of the population in a quiescent state while the rest are active with a time average that can be significantly smaller than the saturation level. These features resemble, to some extent, the observed patterns of neural activity in the cortex, in experiments of short-term memory tasks. The maximal value of P for which stable retrieval phases exist, scales as ${f}^{\mathrm{\ensuremath{-}}3}$/\ensuremath{\Vert}lnf\ensuremath{\Vert} for f\ensuremath{\gg}1/lnN, and as ${f}^{\mathrm{\ensuremath{-}}2}$ln(Nf/\ensuremath{\Vert}lnf\ensuremath{\Vert}) for f\ensuremath{\ll}1/lnN. Numerical simulations of the model with N=1000 and f=0.04 are presented. We also discuss the possible realization of the model in a biologically plausible architecture, where the inhibition is provided by special inhibitory neurons.
---
paper_title: Quality and efficiency of retrieval for Willshaw-like autoassociative networks. II. Recognition

paper_content:

We study the recognition properties of a modification of the Willshaw associative memory with a floating threshold analytically for the asymptotic case of large networks and single-step retrieval. The number of spurious stable states and the size of their attraction basins are calculated. Comparison is made with simulations for single- and multistep dynamics. The appropriateness of the recognition procedure based on the remanent overlap is demonstrated.
---
paper_title: Neural Associative Memories as Accelerators for Binary Vector Search

paper_content:

Associative memories aim at matching an input noisy vector with a stored one. The matched vector satisfies a minimum distance criterion with respect to the inner metric of the device. This problem of finding nearest neighbors in terms of Euclidean or Hamming distances is a very common operation in machine learning and pattern recognition. However, the inner metrics of associative memories are often misfitted to handle practical scenarios. In this paper, we adapt Willshaw networks in order to use them for accelerating nearest neighbor search with limited impact on accuracy. We provide a theoretical analysis of our method for binary sparse vectors. We also test our method using the MNIST handwritten digits database. Both our analysis for synthetic data and experiments with real-data evidence a significant gain in complexity with negligible loss in performance compared to exhaustive search.
---
paper_title: Associative Memories to Accelerate Approximate Nearest Neighbor Search

paper_content:

Nearest neighbor search is a very active field in machine learning for it appears in many application cases, including classification and object retrieval. In its canonical version, the complexity of the search is linear with both the dimension and the cardinal of the collection of vectors the search is performed in. Recently many works have focused on reducing the dimension of vectors using quantization techniques or hashing, while providing an approximate result. In this paper we focus instead on tackling the cardinal of the collection of vectors. Namely, we introduce a technique that partitions the collection of vectors and stores each part in its own associative memory. When a query vector is given to the system, associative memories are polled to identify which one contain the closest match. Then an exhaustive search is conducted only on the part of vectors stored in the selected associative memory. We study the effectiveness of the system when messages to store are generated from i.i.d. uniform $\pm$1 random variables or 0-1 sparse i.i.d. random variables. We also conduct experiment on both synthetic data and real data and show it is possible to achieve interesting trade-offs between complexity and accuracy.
---
paper_title: Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states. Network

paper_content:

A new access to the asymptotic analysis of autoassociation properties in recurrent McCulloch-Pitts networks in the range of low activity is proposed. Using information theory, this method examines the static structure of stable states imprinted by a Hebbian storing process. In addition to the definition of critical pattern capacity usually considered in the analysis of the Hopfield model, the authors introduce the definition of information capacity which guarantees content addressability and is a stricter upper bound of the information really accessible in an autoassociation process. They calculate these two types of capacities for two types of local learning rules which are very effective for sparsely coded patterns: the Hebb rule and the clipped Hebb rule. It turns out that for both rules the information capacity is exactly half the pattern capacity.
---
paper_title: Iterative retrieval of sparsely coded associative memory patterns

paper_content:

Abstract We investigate the pattern completion performance of neural auto-associative memories composed of binary threshold neurons for sparsely coded binary memory patterns. By focussing on iterative retrieval, we are able to introduce effective threshold control strategies. These are investigated by means of computer simulation experiments and analytical treatment. To evaluate the systems performance we consider the completion capacity C and the mean retrieval errors. The asymptotic completion capacity values for the recall of sparsely coded binary patterns in one-step retrieval is known to be ln 2 4 ≈ 17.32% for binary Hebbian learning, and 1 (8 ln 2) ≈ 18% for additive Hebbian learning. These values are accomplished with vanishing error probability and yet are higher than those obtained in other known neural memory models. Recent investigations on binary Hebbian learning have proved that iterative retrieval as a more refined retrieval method does not improve the asymptotic completion capacity of one step retrieval. In a finite size auto-associative memory we show that iterative retrieval achieves higher capacity and better error correction than one-step retrieval. One-step retrieval produces high retrieval errors at optimal memory load. Iterative retrieval reduces the retrieval errors within a few iteration steps (t ⩽ 5). Experiments with additive Hebbian learning show that in the finite model, binary Hebbian learning exhibits much better performance. Thus the main concern of this paper is binary Hebbian learning. We examine iterative retrieval in experiments with up to n = 20,000 threshold neurons. With this system size one-step retrieval yields a completion capacity of about 16%, the second retrieval step increases this value to 17.9% and with iterative retrieval we obtain 19%. The first two retrieval steps in the finite system have also been treated analytically. For one-step retrieval the asymptotic capacity value is approximated from below with growing system size. In the second retrieval step (and as the experiments suggest also for iterative retrieval) the finite size behaviour is different. The capacity exceeds the asymptotic value, reaches an optimum for finite system size, and decreases to the asymptotic limit.
---
paper_title: Learning and pattern recognition in spin glass models

paper_content:

Spin glass models have a complex phase space which may be used to store information. By an asynchronous relaxational dynamics noisy patterns are recognized very fast. In particular the Hopfield model which may be a simple model for neural networks is analyzed in detail. By numerical simulations and analytical approximations the number of patterns to be stored and the amount of noise to be recognized is calculated. It is demonstrated that a random network can learn patterns by reducing frustrated bonds.
---
paper_title: Characteristics of sparsely encoded associative memory

paper_content:

Abstract Characteristics of an autocorrelation or crosscorrelation associative memory largely depend on how items are encoded in pattern vectors to be stored. When most of the components of encoded patterns to be stored are 0 and only a small ratio of the components are 1, the encoding scheme is said to be sparse. The memory capacity and information capacity of a sparsely encoded associative memory are analyzed in detail, and are proved to be in proportion of n 2 log n 2 , n being the number of neurons, which is very large compared to the ordinary non-sparse encoding scheme of about 0.15n. Moreover, it is proved that the sparsely encoded associative memory has a large basin of attraction around each memorized pattern, when and only when an activity control mechanism is attached to it.
---
paper_title: Memory Capacities for Synaptic and Structural Plasticity

paper_content:

Neural associative networks with plastic synapses have been proposed as computational models of brain functions and also for applications such as pattern recognition and information retrieval. To guide biological models and optimize technical applications, several definitions of memory capacity have been used to measure the efficiency of associative memory. Here we explain why the currently used performance measures bias the comparison between models and cannot serve as a theoretical benchmark. We introduce fair measures for information-theoretic capacity in associative memory that also provide a theoretical benchmark. In neural networks, two types of manipulating synapses can be discerned: synaptic plasticity, the change in strength of existing synapses, and structural plasticity, the creation and pruning of synapses. One of the new types of memory capacity we introduce permits quantifying how structural plasticity can increase the network efficiency by compressing the network structure, for example, by pruning unused synapses. Specifically, we analyze operating regimes in the Willshaw model in which structural plasticity can compress the network structure and push performance to the theoretical benchmark. The amount C of information stored in each synapse can scale with the logarithm of the network size rather than being constant, as in classical Willshaw and Hopfield nets (< or = ln 2 approximately 0.7). Further, the review contains novel technical material: a capacity analysis of the Willshaw model that rigorously controls for the level of retrieval quality, an analysis for memories with a nonconstant number of active units (where C < or = 1/e ln 2 approximately 0.53), and the analysis of the computational complexity of associative memories with and without network compression.
---
paper_title: Sparse Neural Networks With Large Learning Diversity

paper_content:

Coded recurrent neural networks with three levels of sparsity are introduced. The first level is related to the size of messages that are much smaller than the number of available neurons. The second one is provided by a particular coding rule, acting as a local constraint in the neural activity. The third one is a characteristic of the low final connection density of the network after the learning phase. Though the proposed network is very simple since it is based on binary neurons and binary connections, it is able to learn a large number of messages and recall them, even in presence of strong erasures. The performance of the network is assessed as a classifier and as an associative memory.
---
paper_title: A Comparative Study of Sparse Associative Memories

paper_content:

We study various models of associative memories with sparse information, i.e. a pattern to be stored is a random string of 0s and 1s with about \(\log N\) 1s, only. We compare different synaptic weights, architectures and retrieval mechanisms to shed light on the influence of the various parameters on the storage capacity.
---
paper_title: Storing Sparse Messages in Networks of Neural Cliques

paper_content:

An extension to a recently introduced binary neural network is proposed to allow the storage of sparse messages, in large numbers and with high memory efficiency. This new network is justified both in biological and informational terms. The storage and retrieval rules are detailed and illustrated by various simulation results.
---
paper_title: Hardware Implementation of Associative Memories Based on Multiple-Valued Sparse Clustered Networks

paper_content:

This paper presents algorithms and hardware implementations of associative memories based on multiple-valued sparse clustered networks (MV-SCNs). SCNs are recently-introduced binary-weighted associative memories that significantly improve the storage and retrieval capabilities over the prior state-of-the art. However, deleting or updating the messages stored in binary-weighted connections result in a significant increase in the data retrieval error probability as the binary-weighted connections deleted may be shared for several data patterns. In order to address the problem, the proposed algorithm exploits multiple-valued weighted connections of the network for storing the messages while maintaining the number of computation nodes in a cluster. The use of the multiple-valued weighted connections reduces the probability of deleting the shared connections compared to the binary-weighted connections. As a result, the proposed algorithm lowers the error rate by an order of magnitude for our sample network with 60% deleted contents compared to the conventional algorithm when the same amount of memory is used. For performance comparisons in hardware, the proposed SCNs are designed using Verilog-HDL and synthesized on TSMC 65 nm CMOS technology. The synthesis results show that the proposed MV-SCNs are around 10% smaller than the conventional binary-weighted SCNs as the number of computation nodes in the proposed SCNs is smaller than that of the conventional SCNs with the comparable speed and memory size.
---
paper_title: Finding All Matches in a Database using Binary Neural Networks

paper_content:

The most efficient architectures of associative memories are based on binary neural networks. As example, Sparse Clustered Networks (SCNs) are able to achieve almost optimal memory efficiency while providing robust indexation of pieces of information through cliques in a neural network. In the canonical formulation of the associative memory problem, the unique stored message matching a given input probe is to be retrieved. In this paper, we focus on the more general problem of finding all messages matching the given probe. We consider real datasets from which many different messages can match given probes, which cannot be done with uniformly distributed messages due to their unlikelyhood of sharing large common parts with one another. Namely, we implement a crossword dictionary containing 8-letter english words, and a chess endgame dataset using associative memories based on binary neural networks. We explain how to adapt SCNs' architecture to this challenging dataset and introduce a backtracking procedure to retrieve all completions of the given input. We stress the performance of the proposed method using different measures and discuss the importance of parameters.
---
paper_title: Quality and efficiency of retrieval for Willshaw-like autoassociative networks: III. Willshaw–Potts model

paper_content:

We study the recognition and correction properties of a structured associative memory with a floating threshold which can be considered a modification of the Willshaw network and bears some resemblance to a network of Potts neurons. An analytical study is performed for the asymptotic case of large networks and single-step retrieval. The main informational characteristics of the network are obtained; the number of spurious stable states and the size of their attraction basins are calculated. A comparison is made with the Willshaw network considered earlier in this series of papers.
---
paper_title: Vector neural net identifying many strongly distorted and correlated patterns

paper_content:

Abstract We suggest an effective and simple algorithm providing a polynomial storage capacity of a network of the form M ~ N 2 s+1 , where N is the dimension of the stored binary patterns. In this problem the value of the free parameter s is restricted by the inequalities N >> s ln N 1 . The algorithm allows us to identify a large number of highly distorted similar patterns. The negative influence of correlations of the patterns is suppressed by choosing a sufficiently large value of the parameter s. We show the efficiency of the algorithm by the example of a perceptron identifier, but it also can be used to increase the storage capacity of full connected systems of associative memory. 1.Introduction Vector models of neural networks were investigated in a lot of papers [1-11]. Among them the most well-known is the Potts spin-glass model [4]. Its properties were investigated rather well by means of the statistical physics methods [5-9]. The characteristics of the memory of the Potts model were analyzed with the aid of computer simulations mainly. In [1-3], as well as in the series of the following papers [9-11], the so called parametrical neural network directed at realization in the form of an optical device, was investigated. In the last case rather simple analytical expressions describing its efficiency, storage capacity and noise immunity were obtained. The analysis of vector neural networks showed that they had an extremely large storage capacity, which was much higher than in the widely known binary models of the Hopfield type. In addition they have an extremely large noise immunity and the ability for recognition in the presence of very large distortions. At the present parametrical vector models of the associative memory are the best both with regard to the storage capacity and noise immunity. In the same time such high parameters of the aforementioned models were practically not used up to now, as well as other models of the associative memory. Moreover, for a lot of people the vector associative memory was related to the processing of the colored images only. The situation changed after the publication of the paper [3], where the algorithm of mapping of binary patterns into q-valued ones was proposed. It was also shown in [3] that such mapping allows one to use vector neural networks for storing and processing of signals of any type and any dimension. Moreover, the mapping brings to nothing the main difficulty of all the associative memory systems, which is the negative influence of correlations between the patterns. Thus, in the
---
paper_title: Twin Neurons for Efficient Real-World Data Distribution in Networks of Neural Cliques: Applications in Power Management in Electronic Circuits

paper_content:

Associative memories are data structures that allow retrieval of previously stored messages given part of their content. They, thus, behave similarly to the human brain’s memory that is capable, for instance, of retrieving the end of a song, given its beginning. Among different families of associative memories, sparse ones are known to provide the best efficiency (ratio of the number of bits stored to that of the bits used). Recently, a new family of sparse associative memories achieving almost optimal efficiency has been proposed. Their structure, relying on binary connections and neurons, induces a direct mapping between input messages and stored patterns. Nevertheless, it is well known that nonuniformity of the stored messages can lead to a dramatic decrease in performance. In this paper, we show the impact of nonuniformity on the performance of this recent model, and we exploit the structure of the model to improve its performance in practical applications, where data are not necessarily uniform. In order to approach the performance of networks with uniformly distributed messages presented in theoretical studies, twin neurons are introduced. To assess the adapted model, twin neurons are used with the real-world data to optimize power consumption of electronic circuits in practical test cases.
---
paper_title: Generalization of features in the assembly neural networks.

paper_content:

The purpose of the paper is an experimental study of the formation of class descriptions, taking place during learning, in assembly neural networks. The assembly neural network is artificially partitioned into several sub-networks according to the number of classes that the network has to recognize. The features extracted from input data are represented in neural column structures of the sub-networks. Hebbian neural assemblies are formed in the column structure of the sub-networks by weight adaptation. A specific class description is formed in each sub-network of the assembly neural network due to intersections between the neural assemblies. The process of formation of class descriptions in the sub-networks is interpreted as feature generalization. A set of special experiments is performed to study this process, on a task of character recognition using the MNIST database.
---
paper_title: An Autoassociative Neural Network Model of Paired-Associate Learning

paper_content:

Hebbian heteroassociative learning is inherently asymmetric. Storing a forward association, from item A to item B, enables recall of B (given A), but does not permit recall of A (given B). Recurrent networks can solve this problem by associating A to B and B back to A. In these recurrent networks, the forward and backward associations can be differentially weighted to account for asymmetries in recall performance. In the special case of equal strength forward and backward weights, these recurrent networks can be modeled as a single autoassociative network where A and B are two parts of a single, stored pattern. We analyze a general, recurrent neural network model of associative memory and examine its ability to fit a rich set of experimental data on human associative learning. The model fits the data significantly better when the forward and backward storage strengths are highly correlated than when they are less correlated. This network-based analysis of associative learning supports the view that associations between symbolic elements are better conceptualized as a blending of two ideas into a single unit than as separately modifiable forward and backward associations linking representations in memory.
---
paper_title: Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing

paper_content:

In this paper, we propose a new approach to implementing hierarchical graph neuron (HGN), an architecture for memorizing patterns of generic sensor stimuli, through the use of vector symbolic architectures. The adoption of a vector symbolic representation ensures a single-layer design while retaining the existing performance characteristics of HGN. This approach significantly improves the noise resistance of the HGN architecture, and enables a linear (with respect to the number of stored entries) time search for an arbitrary subpattern.
---
paper_title: Modeling Brain Function: The World of Attractor Neural Networks

paper_content:

From the Publisher: ::: Substantial progress in understanding memory, the learning process, and self-organization by studying the properties of models of neural networks have resulted in discoveries of important parallels between the properties of statistical, nonlinear cooperative systems in physics and neural networks.
---
paper_title: Boolean Factor Analysis by Attractor Neural Network

paper_content:

A common problem encountered in disciplines such as statistics, data analysis, signal processing, textual data representation, and neural network research, is finding a suitable representation of the data in the lower dimension space. One of the principles used for this reason is a factor analysis. In this paper, we show that Hebbian learning and a Hopfield-like neural network could be used for a natural procedure for Boolean factor analysis. To ensure efficient Boolean factor analysis, we propose our original modification not only of Hopfield network architecture but also its dynamics as well. In this paper, we describe neural network implementation of the Boolean factor analysis method. We show the advantages of our Hopfield-like network modification step by step on artificially generated data. At the end, we show the efficiency of the method on artificial data containing a known list of factors. Our approach has the advantage of being able to analyze very large data sets while preserving the nature of the data
---
paper_title: Psychological concepts in a parallel system

paper_content:

Abstract : Many parallel information processing systems have ben proposed which are loosely based on architecture of the nervous system. Many of the testable predictions of these systems fall in the realm of cognitive science since they perform some 'computations' well and some poorly and have pronounced 'psychologies'. In such system, simple elements are connected to each other; the elements act like 'neurons' in that they sum excitation and inhibition from other elements. Information is represented as large state vectors of element activities, and it can be shown that simple 'synaptic' learing rules can serve to associate arbitrary state vectors using a matrix of connections. One version of this class of models has been shown,, in the past to form concepts and to perform classification computations. This paper discusses the general importance and structure of psychological concepts and describes three simulations of a simple neural model that attempts to reproduce a few of the important properties of human concept formation. Prototypes; Neural Models; Neural nets.
---
paper_title: A distributed representation of temporal context

paper_content:

The principles of recency and contiguity are two cornerstones of the theoretical and empirical analysis of human memory. Recency has been alternatively explained by mechanisms of decay, displacement, and retroactive interference. Another account of recency is based on the idea of variable context (Estes, 1955; Mensink & Raaijmakers, 1989). Such notions are typically cast in terms of a randomly fluctuating population of elements reflective of subtle changes in the environment or in the subjects' mental state. This random context view has recently been incorporated into distributed and neural network memory models (Murdock, 1997; Murdock, Smith, & Bai, 2001). Here we propose an alternative model. Rather than being driven by random fluctuations, this formulation, the temporal context model (TCM), uses retrieval of prior contextual states to drive contextual drift. In TCM, retrieved context is an inherently asymmetric retrieval cue. This allows the model to provide a principled explanation of the widespread advantage for forward recalls in free and serial recall. Modeling data from single-trial free recall, we demonstrate that TCM can simultaneously explain recency and contiguity effects across time scales.
---
paper_title: Vector Symbolic Architectures: A New Building Material for Artificial General Intelligence 1

paper_content:

We provide an overview of Vector Symbolic Architectures (VSA), a class of structured associative memory models that offers a number of desirable features for artificial general intelligence. By directly encoding structure using familiar, computationally efficient algorithms, VSA bypasses many of the problems that have consumed unnecessary effort and attention in previous connectionist work. Example applications from opposite ends of the AI spectrum --visual map-seeking circuits and structured analogy processing --attest to the generality and power of the VSA approach in building new solutions for AI.
---
paper_title: Neural networks with hierarchically correlated patterns

paper_content:

The Hopfield model of neural networks is extended to allow for the storage and retrieval of hierarchically correlated patterns. The overlaps between these patterns form an ultrametric tree. Intermediate states, which serve as ancestors for the following levels, are generated at each level of the tree. The states belonging to each level are stored, by a modified learning rule, in a series of identical networks, one for each level. The retrieval of a particular pattern is preceded and assisted by the successive retrieval of its ancestors. The performance of this scheme is studied analytically and numerically.
---
paper_title: Secondary learning in the assembly neural network

paper_content:

A neural network with assembly organization is described. The network is artificially partitioned into several sub-networks according to the number of classes that the network has to recognize. The features extracted from input data are encoded into activation of certain patterns of neurons in the sub-networks. During a process of primary learning, Hebb's neural assemblies are formed in the sub-networks by means of modification of connections' weights. A procedure of secondary learning, which is named as that of differentiation, is described. The procedure is intended to improve a recognition accuracy of the network by means of additional modification of connections' weights between the neurons of the same sub-networks. A computer simulation of the network is performed. The differentiation process is studied in a set of experiments on a character recognition task using two types of objects: Ukrainian letters and Arabic numerals of modified US National Institute of Standards and Technology (MNIST) database.
---
paper_title: A neural model of the dynamic activation of memory

paper_content:

We study an Attractor Neural Network that stores natural concepts, organized in semantic classes. The concepts are represented by distributed patterns over a space of attributes, and are related by both semantic and episodic associations. While semantic relations are expressed through an hierarchical coding over the attribute space, episodic links are realized via specific synaptic projections. Due to dynamic thresholds expressing neuronal fatigue, the network's behavior is characterized by convergence toward the concept patterns on a short time scale, and by transitions between the various patterns on a longer time scale. In its baseline, undamaged state, the network manifests semantic, episodic, and random transitions, and demonstrates the phenomen of priming. Modeling possible pathological changes, we have found that increasing the ‘noise’ level or the rate of neuronal fatigue decreases the frequency of semantic transitions. When neurons characterized by large synaptic connectivity are deleted, semantic transitions decay before the episodic ones, in accordance with the findings in patients with Alzheimer's disease.
---
paper_title: Modality classification of medical images with distributed representations based on cellular automata reservoir computing

paper_content:

Modality corresponding to medical images is a vital filter in medical image retrieval systems. This article presents the classification of modalities of medical images based on the usage of principles of hyper-dimensional computing and reservoir computing. It is demonstrated that the highest classification accuracy of the proposed method is on a par with the best classical method for the given dataset (83% vs. 84%). The major positive property of the proposed method is that it does not require any optimization routine during the training phase and naturally allows for incremental learning upon the availability of new training data.
---
paper_title: Similarity-Based Retrieval with Structure-Sensitive Sparse Binary Distributed Representations

paper_content:

We present an approach to similarity-based retrieval from knowledge bases that takes into account both the structure and semantics of knowledge base fragments. Those fragments, or analogues, are represented as sparse binary vectors that allow a computationally efficient estimation of structural and semantic similarity by the vector dot product. We present the representation scheme and experimental results for the knowledge base that was previously used for testing of leading analogical retrieval models MAC/FAC and ARCS. The experiments show that the proposed single-stage approach provides results compatible with or better than the results of two-stage models MAC/FAC and ARCS in terms of recall and precision. We argue that the proposed representation scheme is useful for large-scale knowledge bases and free-structured database applications.
---
paper_title: Scaling Laws of Associative Memory Retrieval

paper_content:

Most people have great difficulty in recalling unrelated items. For example, in free recall experiments, lists of more than a few randomly selected words cannot be accurately repeated. Here we introduce a phenomenological model of memory retrieval inspired by theories of neuronal population coding of information. The model predicts nontrivial scaling behaviors for the mean and standard deviation of the number of recalled words for lists of increasing length. Our results suggest that associative information retrieval is a dominating factor that limits the number of recalled items.
---
paper_title: Distinctive features, categorical perception, and probability learning: some applications of a neural model

paper_content:

A previously proposed model for memory based on neurophysiolo gical considerations is reviewed. We assume that (a) nervous system activity is usefully represented as the set of simultaneous individual neuron activities in a group of neurons; (b) different memory traces make use of the same synapses; and (c) synapses associate two patterns of neural activity by incrementing synaptic connectivity proportionally to the product of pre- and postsynaptic activity, forming a matrix of synaptic connectivities. We extend this model by (a) introducing positive feedback of a set of neurons onto itself and (b) allowing the individual neurons to saturate. A hybrid model, partly analog and partly binary, arises. The system has certain characteristics reminiscent of analysis by distinctive features. Next, we apply the model to "categorical perception." Finally, we discuss probability learning. The model can predict overshooting, recency data, and probabilities occurring in systems with more than two events with reasonably good accuracy.
---
paper_title: Recurrent-Neural-Network-Based Boolean Factor Analysis and Its Application to Word Clustering

paper_content:

The objective of this paper is to introduce a neural-network-based algorithm for word clustering as an extension of the neural-network-based Boolean factor analysis algorithm (Frolov , 2007). It is shown that this extended algorithm supports even the more complex model of signals that are supposed to be related to textual documents. It is hypothesized that every topic in textual data is characterized by a set of words which coherently appear in documents dedicated to a given topic. The appearance of each word in a document is coded by the activity of a particular neuron. In accordance with the Hebbian learning rule implemented in the network, sets of coherently appearing words (treated as factors) create tightly connected groups of neurons, hence, revealing them as attractors of the network dynamics. The found factors are eliminated from the network memory by the Hebbian unlearning rule facilitating the search of other factors. Topics related to the found sets of words can be identified based on the words' semantics. To make the method complete, a special technique based on a Bayesian procedure has been developed for the following purposes: first, to provide a complete description of factors in terms of component probability, and second, to enhance the accuracy of classification of signals to determine whether it contains the factor. Since it is assumed that every word may possibly contribute to several topics, the proposed method might be related to the method of fuzzy clustering. In this paper, we show that the results of Boolean factor analysis and fuzzy clustering are not contradictory, but complementary. To demonstrate the capabilities of this attempt, the method is applied to two types of textual data on neural networks in two different languages. The obtained topics and corresponding words are at a good level of agreement despite the fact that identical topics in Russian and English conferences contain different sets of keywords.
---
paper_title: Slow stochastic Hebbian learning of classes of stimuli in a recurrent neural network

paper_content:

We study unsupervised Hebbian learning in a recurrent network in which synapses have a finite number of stable states. Stimuli received by the network are drawn at random at each presentation from a set of classes. Each class is defined as a cluster in stimulus space, centred on the class prototype. The presentation protocol is chosen to mimic the protocols of visual memory experiments in which a set of stimuli is presented repeatedly in a random way. The statistics of the input stream may be stationary, or changing. Each stimulus induces, in a stochastic way, transitions between stable synaptic states. Learning dynamics is studied analytically in the slow learning limit, in which a given stimulus has to be presented many times before it is memorized, i.e. before synaptic modifications enable a pattern of activity correlated with the stimulus to become an attractor of the recurrent network. We show that in this limit the synaptic matrix becomes more correlated with the class prototypes than with any of the instances of the class. We also show that the number of classes that can be learned increases sharply when the coding level decreases, and determine the speeds of learning and forgetting of classes in the case of changes in the statistics of the input stream.
---
paper_title: Binary Vectors for Fast Distance and Similarity Estimation

paper_content:

This review considers methods and algorithms for fast estimation of distance/similarity measures between initial data from vector representations with binary or integer-valued components obtained from initial data that are mainly high-dimensional vectors with different distance measures (angular, Euclidean, and others) and similarity measures (cosine, inner product, and others). Methods without learning that mainly use random projections with the subsequent quantization and also sampling methods are discussed. The obtained vectors can be applied in similarity search, machine learning, and other algorithms.
---
paper_title: Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors

paper_content:

The 1990s saw the emergence of cognitive models that depend on very high dimensionality and randomness. They include Holographic Reduced Representations, Spatter Code, Semantic Vectors, Latent Semantic Analysis, Context-Dependent Thinning, and Vector-Symbolic Architecture. They represent things in high-dimensional vectors that are manipulated by operations that produce new high-dimensional vectors in the style of traditional computing, in what is called here hyperdimensional computing on account of the very high dimensionality. The paper presents the main ideas behind these models, written as a tutorial essay in hopes of making the ideas accessible and even provocative. A sketch of how we have arrived at these models, with references and pointers to further reading, is given at the end. The thesis of the paper is that hyperdimensional representation has much to offer to students of cognitive science, theoretical neuroscience, computer science and engineering, and mathematics.
---
paper_title: An assembly neural network for texture segmentation

paper_content:

Abstract An architecture of a neural network with assembly organization is described. Such network architecture is applied to the problem of texture segmentation in natural scenes. The network is partitioned into several subnetworks. Each subnetwork is a column structure in which features are represented by means of “float” coding. Input data excite corresponding “floats” of neurons in the subnetworks. In the process of learning the weights of modifiable connections between excited neurons are changed so that Hebb's assemblies are formed in the column structures. All subnetworks are incorporated into a single network by a neural activity control system. Computer simulation of the proposed network has been performed. The results of computer simulations show the possibility of successful application of the assembly neural network to the problem of texture segmentation.
---
paper_title: Recognizing Permuted Words with Vector Symbolic Architectures: A Cambridge Test for Machines

paper_content:

Abstract This paper proposes a simple encoding scheme for words using principles of Vector Symbolic Architectures. The proposed encoding allows finding a valid word in the dictionary for a given permuted word (represented using the proposed approach) using only a single operation– calculation of Hamming distance to the distributed representations of valid words in the dictionary. The proposed encoding scheme can be used as an additional processing mechanism for models of word embedding, which also form vectors to represent the meanings of words, in order to match the distorted words in the text to the valid words in the dictionary.
---
paper_title: Building a world model with structure-sensitive sparse binary distributed representations

paper_content:

We present a new cognitive architecture named Associative-Projective Neural Networks (APNNs). APNNs have a multi-module, multi-level, and multi-modal design that works with an original scheme of sparse binary distributed representations to construct world models of varied complexity required for both task-specific and more general cognitive modeling. APNNs provide scalability and flexibility due to a number of design features. Internal representations of APNNs are sparse binary vectors of fixed dimensionality for items of various complexity and generality. Representations of input scalars, vectors, or compositional relational structures are constructed on-the-fly, so that similar items produce representations similar in terms of vector dot-products. Thus, for example, similarity of relational structures (taking into account similarity of their components, their grouping and order) can be estimated by dot-products of their representations, without the need to follow edges or to match vertices of underlying graphs. Decoding distributed representations through the input representations is also possible. Storage, retrieval, and decoding of distributed representations are implemented by efficient auto-associative memories; using distributed memories based on the idea of Hebb’s cell assemblies additionally provides a natural tool for emergence of generalization hierarchies. In addition, we consider how APNNs account for representation grounding, deal with recent challenges for distributed representations, and present some open problems.
---
paper_title: Modular neural networks with Hebbian learning rule

paper_content:

The paper consists of two parts, each of them describing a learning neural network with the same modular architecture and with a similar set of functioning algorithms. Both networks are artificially partitioned into several equal modules according to the number of classes that the network has to recognize. Hebbian learning rule is used for network training. In the first network, learning process is concentrated inside the modules so that a system of intersecting neural assemblies is formed in each module. Unlike that, in the second network, learning connections link only neurons of different modules. Computer simulation of the networks is performed. Testing of the networks is executed on the MNIST database. Both networks directly use brightness values of image pixels as features. The second network has a better performance than the first one and demonstrates the recognition rate of 98.15%.
---
paper_title: Representation and processing of structures with binary sparse distributed codes

paper_content:

The schemes for compositional distributed representations include those allowing on-the-fly construction of fixed dimensionality codevectors to encode structures of various complexity. Similarity of such codevectors takes into account both structural and semantic similarity of represented structures. We provide a comparative description of sparse binary distributed representation developed in the framework of the associative-projective neural network architecture and the more well known holographic reduced representations of T.A. Plate (1995) and binary spatter codes of P. Kanerva (1996). The key procedure in associative-projective neural networks is context-dependent thinning which binds codevectors and maintains their sparseness. The codevectors are stored in structured memory array which can be realized as distributed auto-associative memory. Examples of distributed representation of structured data are given. Fast estimation of the similarity of analogical episodes by the overlap of their codevectors is used in the modeling of analogical reasoning both for retrieval of analogs from memory and for analogical mapping.
---
paper_title: On bidirectional transitions between localist and distributed representations : The case of common substrings search using Vector Symbolic Architecture

paper_content:

The contribution of this article is twofold. First, it presents an encoding approach for seamless bidirectional transitions between localist and distributed representation domains. Second, the appr ...
---
paper_title: THE AUGMENTED MODELS OF ASSOCIATIVE MEMORY ASYMMETRIC INTERACTION AND HIERARCHY OF PATTERNS

paper_content:

The asymmetric modification of the Hopfield model of the associative memory is considered. It is shown that the asymmetry does not change the main properties of the model, but leads to the internal nonthermal noise. The modification of the Hopfield algorithm is proposed which can be used for storing the correlated patterns and its storage capacity is estimated. The hierarchical memory model is proposed and studied.
---
paper_title: ‘Unlearning’ has a stabilizing effect in collective memories

paper_content:

Crick and Mitchison1 have presented a hypothesis for the functional role of dream sleep involving an ‘unlearning’ process. We have independently carried out mathematical and computer modelling of learning and ‘unlearning’ in a collective neural network of 30–1,000 neurones. The model network has a content-addressable memory or ‘associative memory’ which allows it to learn and store many memories. A particular memory can be evoked in its entirety when the network is stimulated by any adequate-sized subpart of the information of that memory2. But different memories of the same size are not equally easy to recall. Also, when memories are learned, spurious memories are also created and can also be evoked. Applying an ‘unlearning’ process, similar to the learning processes but with a reversed sign and starting from a noise input, enhances the performance of the network in accessing real memories and in minimizing spurious ones. Although our model was not motivated by higher nervous function, our system displays behaviours which are strikingly parallel to those needed for the hypothesized role of ‘unlearning’ in rapid eye movement (REM) sleep.
---
paper_title: Some approaches to analogical mapping with structure-sensitive distributed representations

paper_content:

This paper presents some techniques for analogical mapping using associative-projective neural networks (APNNs). Sparse binary distributed representations of constant high dimensionality are constructed on-the-fly for hierarchical structures of various complexity. Such representations encode both surface and structural similarity of analogical episodes. The introduced mapping approaches are illustrated using test analogies.
---
paper_title: HIERARCHICAL ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH LOW ACTIVITY LEVEL

paper_content:

A Hopfield-like neural network that can store hierarchically correlated patterns with low level of activity is studied. Three learning rules are proposed which enable to obtain nearly optimal storage capacity. These learning rules have different rate of biological relevancy and the restrictions they put upon the structure of hierarchical tree. By varying the value of the neural threshold, it is possible to climb up and down the hierarchical tree.
---
paper_title: Fault detection in the hyperspace: Towards intelligent automation systems

paper_content:

This article presents a methodology for intelligent, biologically inspired fault detection system for generic complex systems of systems. The proposed methodology utilizes the concepts of associative memory and vector symbolic architectures, commonly used for modeling cognitive abilities of human brain. Compared to classical methods of artificial intelligence used in the context of fault detection the proposed methodology shows an unprecedented performance, while featuring zero configuration and simple operations.
---
paper_title: The Organization of Behavior: A Neuropsychological Theory

paper_content:

Contents: Introduction. The Problem and the Line of Attack. Summation and Learning in Perception. Field Theory and Equipotentiality. The First Stage of Perception: Growth of the Assembly. Perception of a Complex: The Phase Sequence. Development of the Learning Capacity. Higher and Lower Processes Related to Learning. Problems of Motivation: Pain and Hunger. The Problem of Motivational Drift. Emotional Disturbances. The Growth and Decline of Intelligence.
---
paper_title: Dependable MAC Layer Architecture Based on Holographic Data Representation Using Hyper-Dimensional Binary Spatter Codes

paper_content:

In this article we propose the usage of binary spatter codes and distributed data representation for communicating loss and delay sensitive data in event-driven sensor and actuator networks. Using the proposed data representation technique along with the medium access control protocol the mission critical control information can be transmitted with assured constant delay in deployments exposing below 0 dB signal-to-noise ratio figures.
---
paper_title: Hierarchical Concept Formation in Associative Memory Composed of Neuro-window Elements.

paper_content:

In the present letter, the authors propose a model of hierarchical concept formation in auto-associative memory composed of neuro-window elements. When hierarchically correlated patterns are memorized, each center of the correlated memory patterns can be defined as a concept pattern which represents those similar patterns. It is shown that the concept patterns are formed under a Hebbian learning rule in the neural network where all the neuro-window elements are just mutually connected, and that the memory patterns and the concept patterns in each level of hierarchy can be retrieved selectively by adjusting the parameters of the neuro-window elements. Copyright 1996 Elsevier Science Ltd
---
paper_title: Brain-like classifier of temporal patterns

paper_content:

In this article we present a pattern classification system which uses Vector Symbolic Architecture (VSA) for representation, learning and subsequent classification of patterns, as a showcase we have used classification of vibration sensors measurements to vehicles types. On the quantitative side the proposed classifier requires only 1 kB of memory to classify an incoming signal against of several hundred of training samples. The classification operation into N types requires only 2∗N+1 arithmetic operations this makes the proposed classifier feasible for implementation on a low-end sensor nodes. The main contribution of this article is the proposed methodology for representing temporal patterns with distributed representation and VSA-based classifier.
---
paper_title: Structure of neural assembly

paper_content:

The process of neural assembly formation in neural networks with a low level of activity is investigated. The experiments conducted show the possibility of forming neural assemblies with nucleus and fringe and of restoring both the nucleus and the nucleus with fringe. This proves that the neural network has additional functions: to form prototypes and to restore the assembly nucleus and assembly fringes from their parts. The tests performed have addressed the problem of decreasing neural network capacity. >
---
paper_title: Holographic Reduced Representation: Distributed Representation for Cognitive Structures

paper_content:

While neuroscientists garner success in identifying brain regions and in analyzing individual neurons, ground is still being broken at the intermediate scale of understanding how neurons combine to encode information. This book proposes a method of representing information in a computer that would be suited for modelling the brain's methods of processing information. Holographic reduced representations (HRRs) are introduced here to model how the brain distributes each piece of information among thousands of neurons. It has been previously thought that the grammatical structure of a language cannot be encoded practically in a distributed representation, but HRRs can overcome the problems of earlier proposals. Thus this work has implications for psychology, neuroscience, linguistics, computer science and engineering.
---
paper_title: Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing

paper_content:

In this paper, we propose a new approach to implementing hierarchical graph neuron (HGN), an architecture for memorizing patterns of generic sensor stimuli, through the use of vector symbolic architectures. The adoption of a vector symbolic representation ensures a single-layer design while retaining the existing performance characteristics of HGN. This approach significantly improves the noise resistance of the HGN architecture, and enables a linear (with respect to the number of stored entries) time search for an arbitrary subpattern.
---
paper_title: Memory Vectors for Similarity Search in High-Dimensional Spaces

paper_content:

We study an indexing architecture to store and search in a database of high-dimensional vectors from the perspective of statistical signal processing and decision theory. This architecture is composed of several memory units, each of which summarizes a fraction of the database by a single representative vector. The potential similarity of the query to one of the vectors stored in the memory unit is gauged by a simple correlation with the memory unit's representative vector. This representative optimizes the test of the following hypothesis: the query is independent from any vector in the memory unit versus the query is a simple perturbation of one of the stored vectors. Compared to exhaustive search, our approach finds the most similar database vectors significantly faster without a noticeable reduction in search quality. Interestingly, the reduction of complexity is provably better in high-dimensional spaces. We empirically demonstrate its practical interest in a large-scale image search scenario with off-the-shelf state-of-the-art descriptors.
---
paper_title: Storage capacity of generalized networks

paper_content:

We discuss the thermodynamic phases and storage capacity of models which are extensions of the Hopfield-Little model of associative memory, generalized to allow for arbitrary polynomial Hamiltonians. The storage capacity increases as an exponential function of the highest power appearing in the polynomial Hamiltonian. We include the effects of replica symmetry breaking in our computations.
---
paper_title: Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics

paper_content:

Hyperdimensional (HD) computing is a promising paradigm for future intelligent electronic appliances operating at low power. This paper discusses tradeoffs of selecting parameters of binary HD representations when applied to pattern recognition tasks. Particular design choices include density of representations and strategies for mapping data from the original representation. It is demonstrated that for the considered pattern recognition tasks (using synthetic and real-world data) both sparse and dense representations behave nearly identically. This paper also discusses implementation peculiarities which may favor one type of representations over the other. Finally, the capacity of representations of various densities is discussed.
---
paper_title: An approximate dynamic programming approach for improving accuracy of lossy data compression by Bloom filters

paper_content:

Bloom filters are a data structure for storing data in a compressed form. They offer excellent space and time efficiency at the cost of some loss of accuracy (so-called lossy compression). This work presents a yes–no Bloom filter, which as a data structure consisting of two parts: the yes-filter which is a standard Bloom filter and the no-filter which is another Bloom filter whose purpose is to represent those objects that were recognized incorrectly by the yes-filter (that is, to recognize the false positives of the yes-filter). By querying the no-filter after an object has been recognized by the yes-filter, we get a chance of rejecting it, which improves the accuracy of data recognition in comparison with the standard Bloom filter of the same total length. A further increase in accuracy is possible if one chooses objects to include in the no-filter so that the no-filter recognizes as many as possible false positives but no true positives, thus producing the most accurate yes–no Bloom filter among all yes–no Bloom filters. This paper studies how optimization techniques can be used to maximize the number of false positives recognized by the no-filter, with the constraint being that it should recognize no true positives. To achieve this aim, an Integer Linear Program (ILP) is proposed for the optimal selection of false positives. In practice the problem size is normally large leading to intractable optimal solution. Considering the similarity of the ILP with the Multidimensional Knapsack Problem, an Approximate Dynamic Programming (ADP) model is developed making use of a reduced ILP for the value function approximation. Numerical results show the ADP model works best comparing with a number of heuristics as well as the CPLEX built-in solver (B&B), and this is what can be recommended for use in yes–no Bloom filters. In a wider context of the study of lossy compression algorithms, our research is an example showing how the arsenal of optimization methods can be applied to improving the accuracy of compressed data.
---
paper_title: Dense Associative Memory is Robust to Adversarial Inputs

paper_content:

Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish...
---
paper_title: Network Applications of Bloom Filters: A Survey

paper_content:

A Bloom filter is a simple space-efficient randomized data structure for representing a set in order to support membership queries. Bloom filters allow false positives but the space savings often outweigh this drawback when the probability of an error is controlled. Bloom filters have been used in database applications since the 1970s, but only in recent years have they become popular in the networking literature. The aim of this paper is to survey the ways in which Bloom filters have been used and modified in a variety of network problems, with the aim of providing a unified mathematical and practical framework for understanding them and stimulating their use in future applications.
---
paper_title: Multiconnected neural network models

paper_content:

A generalisation of the Hopfield model which includes interactions between p()2) Ising spins is considered. The exact storage capacity behaves as Np-1/2(p-1)! ln N when the number of nodes, N, is large. In the limit p to infinity , the thermodynamics of the model can be solved exactly without using the replica method; at zero temperature, a solution which is completely correlated with the input pattern exists for alpha < alpha c where alpha c to infinity as p to infinity and this solution has lower energy than the spin-glass solution if alpha < alpha 1=1/4 ln 2 where the number of patterns n=(2 alpha /p!)Np-1. For finite values of p, the correlation with the input pattern is not complete; for p=3 and 4, approximate values of alpha c and alpha 1 are obtained and for p to infinity the replica symmetric approximation gives alpha c approximately p/4 ln p.
---
paper_title: On a model of associative memory with huge storage capacity

paper_content:

In Krotov et al. (in: Lee (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., Red Hook, 2016) Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the ”limit” as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model.
---
paper_title: Distance Sensitive Bloom Filters Without False Negatives

paper_content:

A Bloom filter is a widely used data-structure for representing a set $S$ and answering queries of the form "Is $x$ in $S$?". By allowing some false positive answers (saying "yes" when the answer is in fact `no') Bloom filters use space significantly below what is required for storing $S$. In the distance sensitive setting we work with a set $S$ of (Hamming) vectors and seek a data structure that offers a similar trade-off, but answers queries of the form "Is $x$ close to an element of $S$?" (in Hamming distance). Previous work on distance sensitive Bloom filters have accepted false positive and false negative answers. Absence of false negatives is of critical importance in many applications of Bloom filters, so it is natural to ask if this can be also achieved in the distance sensitive setting. Our main contributions are upper and lower bounds (that are tight in several cases) for space usage in the distance sensitive setting where false negatives are not allowed.
---
paper_title: Long-term attraction in higher order neural networks

paper_content:

Recent results on the memory storage capacity of higher order neural networks indicate a significant improvement compared to the limited capacity of the Hopfield model. However, such results have so far been obtained under the restriction that only a single iteration is allowed to converge. This paper presents a indirect convergence (long-term attraction) analysis of higher order neural networks. Our main result is that for any /spl kappa//sub d/<d!2/sup d-1//(2d)!, and 0/spl les//spl rho/<1/2, a Hebbian higher order neural network of order d with n neurons can store a random set of /spl kappa//sub d/n/sup d//log n fundamental memories such that almost all memories have an attraction radius of size /spl rho/n. If /spl kappa//sub d/<d!2/sup d-1//((2d)!(d+1)), then all memories possess this property simultaneously. It indicates that the lower bounds on the long-term attraction capacities are larger than the corresponding direct convergence capacities by a factor of 1/(1-2/spl rho/)/sup 2d/. In addition we upper bound the convergence rate (number of iterations required to converge). This bound is asymptotically independent of n. Similar results are obtained for zero diagonal higher order neural networks.
---
paper_title: Modification of Holographic Graph Neuron using Sparse Distributed Representations

paper_content:

This article presents a modification of the recently proposed Holographic Graph Neuron approach for memorizing patterns of generic sensor stimuli. The original approach represents patterns as dense ...
---
paper_title: Long term memory storage capacity of multiconnected neural networks

paper_content:

Quantitative expressions of long-term memory storage capacities of complex neural network are derived. The networks are made of neurons connected by synapses of any order, of the axono-axonal type considered by Kandel et al. for example. The effect of link deletion possibly related to aging, is also considered. The central result of this study is that, within the framework of Hebb's laws, the number of stored bits is proportional to the number of synapses. The proportionality factor however, decreases when the order of involved synaptic contact increases. This tends to favor neural architectures with low-order synaptic connectivities. It is finally shown that the memory storage capacities can be optimized by a partition of the network into neuron clusters with size comparable with that observed for cortical microcolumns.
---
paper_title: Building a world model with structure-sensitive sparse binary distributed representations

paper_content:

We present a new cognitive architecture named Associative-Projective Neural Networks (APNNs). APNNs have a multi-module, multi-level, and multi-modal design that works with an original scheme of sparse binary distributed representations to construct world models of varied complexity required for both task-specific and more general cognitive modeling. APNNs provide scalability and flexibility due to a number of design features. Internal representations of APNNs are sparse binary vectors of fixed dimensionality for items of various complexity and generality. Representations of input scalars, vectors, or compositional relational structures are constructed on-the-fly, so that similar items produce representations similar in terms of vector dot-products. Thus, for example, similarity of relational structures (taking into account similarity of their components, their grouping and order) can be estimated by dot-products of their representations, without the need to follow edges or to match vertices of underlying graphs. Decoding distributed representations through the input representations is also possible. Storage, retrieval, and decoding of distributed representations are implemented by efficient auto-associative memories; using distributed memories based on the idea of Hebb’s cell assemblies additionally provides a natural tool for emergence of generalization hierarchies. In addition, we consider how APNNs account for representation grounding, deal with recent challenges for distributed representations, and present some open problems.
---
paper_title: Dense Associative Memory for Pattern Recognition

paper_content:

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.
---
paper_title: Theory and Practice of Bloom Filters for Distributed Systems

paper_content:

Many network solutions and overlay networks utilize probabilistic techniques to reduce information processing and networking costs. This survey article presents a number of frequently used and useful probabilistic techniques. Bloom filters and their variants are of prime importance, and they are heavily used in various distributed systems. This has been reflected in recent research and many new algorithms have been proposed for distributed systems that are either directly or indirectly based on Bloom filters. In this survey, we give an overview of the basic and advanced techniques, reviewing over 20 variants and discussing their application in distributed systems, in particular for caching, peer-to-peer systems, routing and forwarding, and measurement data summarization.
---
paper_title: Holographic Reduced Representation: Distributed Representation for Cognitive Structures

paper_content:

While neuroscientists garner success in identifying brain regions and in analyzing individual neurons, ground is still being broken at the intermediate scale of understanding how neurons combine to encode information. This book proposes a method of representing information in a computer that would be suited for modelling the brain's methods of processing information. Holographic reduced representations (HRRs) are introduced here to model how the brain distributes each piece of information among thousands of neurons. It has been previously thought that the grammatical structure of a language cannot be encoded practically in a distributed representation, but HRRs can overcome the problems of earlier proposals. Thus this work has implications for psychology, neuroscience, linguistics, computer science and engineering.
---
paper_title: Iterative Learning and Denoising in Convolutional Neural Associative Memories

paper_content:

The task of a neural associative memory is to retrieve a set of previously memorized patterns from their noisy versions by using a network of neurons. Hence, an ideal network should be able to 1) gradually learn a set of patterns, 2) retrieve the correct pattern from noisy queries and 3) maximize the number of memorized patterns while maintaining the reliability in responding to queries. We show that by considering the inherent redundancy in the memorized patterns, one can obtain all the mentioned properties at once. This is in sharp contrast with previous work that could only improve one or two aspects at the expense of the others. More specifically, we devise an iterative algorithm that learns the redundancy among the patterns. The resulting network has a retrieval capacity that is exponential in the size of the network. Lastly, by considering the local structures of the network, the asymptotic error correction performance can be made linear in the size of the network.
---
paper_title: Associative Memory using Dictionary Learning and Expander Decoding

paper_content:

An associative memory is a framework of content-addressable memory that stores a collection of message vectors (or a dataset) over a neural network while enabling a neurally feasible mechanism to recover any message in the dataset from its noisy version. Designing an associative memory requires addressing two main tasks: 1) learning phase: given a dataset, learn a concise representation of the dataset in the form of a graphical model (or a neural network), 2) recall phase: given a noisy version of a message vector from the dataset, output the correct message vector via a neurally feasible algorithm over the network learnt during the learning phase. This paper studies the problem of designing a class of neural associative memories which learns a network representation for a large dataset that ensures correction against a large number of adversarial errors during the recall phase. Specifically, the associative memories designed in this paper can store dataset containing $\exp(n)$ $n$-length message vectors over a network with $O(n)$ nodes and can tolerate $\Omega(\frac{n}{{\rm polylog} n})$ adversarial errors. This paper carries out this memory design by mapping the learning phase and recall phase to the tasks of dictionary learning with a square dictionary and iterative error correction in an expander code, respectively.
---
paper_title: Nonbinary Associative Memory With Exponential Pattern Retrieval Capacity and Iterative Learning

paper_content:

We consider the problem of neural association for a network of nonbinary neurons. Here, the task is to first memorize a set of patterns using a network of neurons whose states assume values from a finite number of integer levels. Later, the same network should be able to recall the previously memorized patterns from their noisy versions. Prior work in this area consider storing a finite number of purely random patterns, and have shown that the pattern retrieval capacities (maximum number of patterns that can be memorized) scale only linearly with the number of neurons in the network. In our formulation of the problem, we concentrate on exploiting redundancy and internal structure of the patterns to improve the pattern retrieval capacity. Our first result shows that if the given patterns have a suitable linear-algebraic structure, i.e., comprise a subspace of the set of all possible patterns, then the pattern retrieval capacity is exponential in terms of the number of neurons. The second result extends the previous finding to cases where the patterns have weak minor components, i.e., the smallest eigenvalues of the correlation matrix tend toward zero. We will use these minor components (or the basis vectors of the pattern null space) to increase both the pattern retrieval capacity and error correction capabilities. An iterative algorithm is proposed for the learning phase, and two simple algorithms are presented for the recall phase. Using analytical methods and simulations, we show that the proposed methods can tolerate a fair amount of errors in the input while being able to memorize an exponentially large number of patterns.
---
paper_title: Associative memory via a sparse recovery model

paper_content:

An associative memory is a structure learned from a dataset M of vectors (signals) in a way such that, given a noisy version of one of the vectors as input, the nearest valid vector from M (nearest neighbor) is provided as output, preferably via a fast iterative algorithm. Traditionally, binary (or q-ary) Hopfield neural networks are used to model the above structure. In this paper, for the first time, we propose a model of associative memory based on sparse recovery of signals. Our basic premise is simple. For a dataset, we learn a set of linear constraints that every vector in the dataset must satisfy. Provided these linear constraints possess some special properties, it is possible to cast the task of finding nearest neighbor as a sparse recovery problem. Assuming generic random models for the dataset, we show that it is possible to store super-polynomial or exponential number of n-length vectors in a neural network of size O(n). Furthermore, given a noisy version of one of the stored vectors corrupted in near-linear number of coordinates, the vector can be correctly recalled using a neurally feasible algorithm.
---
paper_title: Iterative Learning and Denoising in Convolutional Neural Associative Memories

paper_content:

The task of a neural associative memory is to retrieve a set of previously memorized patterns from their noisy versions by using a network of neurons. Hence, an ideal network should be able to 1) gradually learn a set of patterns, 2) retrieve the correct pattern from noisy queries and 3) maximize the number of memorized patterns while maintaining the reliability in responding to queries. We show that by considering the inherent redundancy in the memorized patterns, one can obtain all the mentioned properties at once. This is in sharp contrast with previous work that could only improve one or two aspects at the expense of the others. More specifically, we devise an iterative algorithm that learns the redundancy among the patterns. The resulting network has a retrieval capacity that is exponential in the size of the network. Lastly, by considering the local structures of the network, the asymptotic error correction performance can be made linear in the size of the network.
---
paper_title: Associative Memory using Dictionary Learning and Expander Decoding

paper_content:

An associative memory is a framework of content-addressable memory that stores a collection of message vectors (or a dataset) over a neural network while enabling a neurally feasible mechanism to recover any message in the dataset from its noisy version. Designing an associative memory requires addressing two main tasks: 1) learning phase: given a dataset, learn a concise representation of the dataset in the form of a graphical model (or a neural network), 2) recall phase: given a noisy version of a message vector from the dataset, output the correct message vector via a neurally feasible algorithm over the network learnt during the learning phase. This paper studies the problem of designing a class of neural associative memories which learns a network representation for a large dataset that ensures correction against a large number of adversarial errors during the recall phase. Specifically, the associative memories designed in this paper can store dataset containing $\exp(n)$ $n$-length message vectors over a network with $O(n)$ nodes and can tolerate $\Omega(\frac{n}{{\rm polylog} n})$ adversarial errors. This paper carries out this memory design by mapping the learning phase and recall phase to the tasks of dictionary learning with a square dictionary and iterative error correction in an expander code, respectively.
---
paper_title: Nonbinary Associative Memory With Exponential Pattern Retrieval Capacity and Iterative Learning

paper_content:

We consider the problem of neural association for a network of nonbinary neurons. Here, the task is to first memorize a set of patterns using a network of neurons whose states assume values from a finite number of integer levels. Later, the same network should be able to recall the previously memorized patterns from their noisy versions. Prior work in this area consider storing a finite number of purely random patterns, and have shown that the pattern retrieval capacities (maximum number of patterns that can be memorized) scale only linearly with the number of neurons in the network. In our formulation of the problem, we concentrate on exploiting redundancy and internal structure of the patterns to improve the pattern retrieval capacity. Our first result shows that if the given patterns have a suitable linear-algebraic structure, i.e., comprise a subspace of the set of all possible patterns, then the pattern retrieval capacity is exponential in terms of the number of neurons. The second result extends the previous finding to cases where the patterns have weak minor components, i.e., the smallest eigenvalues of the correlation matrix tend toward zero. We will use these minor components (or the basis vectors of the pattern null space) to increase both the pattern retrieval capacity and error correction capabilities. An iterative algorithm is proposed for the learning phase, and two simple algorithms are presented for the recall phase. Using analytical methods and simulations, we show that the proposed methods can tolerate a fair amount of errors in the input while being able to memorize an exponentially large number of patterns.
---
paper_title: Associative memory via a sparse recovery model

paper_content:

An associative memory is a structure learned from a dataset M of vectors (signals) in a way such that, given a noisy version of one of the vectors as input, the nearest valid vector from M (nearest neighbor) is provided as output, preferably via a fast iterative algorithm. Traditionally, binary (or q-ary) Hopfield neural networks are used to model the above structure. In this paper, for the first time, we propose a model of associative memory based on sparse recovery of signals. Our basic premise is simple. For a dataset, we learn a set of linear constraints that every vector in the dataset must satisfy. Provided these linear constraints possess some special properties, it is possible to cast the task of finding nearest neighbor as a sparse recovery problem. Assuming generic random models for the dataset, we show that it is possible to store super-polynomial or exponential number of n-length vectors in a neural network of size O(n). Furthermore, given a noisy version of one of the stored vectors corrupted in near-linear number of coordinates, the vector can be correctly recalled using a neurally feasible algorithm.
---
paper_title: Boolean Factor Analysis by Attractor Neural Network

paper_content:

A common problem encountered in disciplines such as statistics, data analysis, signal processing, textual data representation, and neural network research, is finding a suitable representation of the data in the lower dimension space. One of the principles used for this reason is a factor analysis. In this paper, we show that Hebbian learning and a Hopfield-like neural network could be used for a natural procedure for Boolean factor analysis. To ensure efficient Boolean factor analysis, we propose our original modification not only of Hopfield network architecture but also its dynamics as well. In this paper, we describe neural network implementation of the Boolean factor analysis method. We show the advantages of our Hopfield-like network modification step by step on artificially generated data. At the end, we show the efficiency of the method on artificial data containing a known list of factors. Our approach has the advantage of being able to analyze very large data sets while preserving the nature of the data
---
paper_title: Dictionary matching and indexing with errors and don't cares

paper_content:

This paper considers various flavors of the following online problem: preprocess a text or collection of strings, so that given a query string p, all matches of p with the text can be reported quickly. In this paper we consider matches in which a bounded number of mismatches are allowed, or in which a bounded number of "don't care" characters are allowed. The specific problems we look at are: indexing, in which there is a single text t, and we seek locations where p matches a substring of t; dictionary queries, in which a collection of strings is given upfront, and we seek those strings which match p in their entirety; and dictionary matching, in which a collection of strings is given upfront, and we seek those substrings of a (long) p which match an original string in its entirety. These are all instances of an all-to-all matching problem, for which we provide a single solution.The performance bounds all have a similar character. For example, for the indexing problem with n=|t| and m=|p|, the query time for k substitutions is O(m + (c 1 log n)k⁄k! + # matches), with a data structure of size O(n (c 2 log n)k⁄k!) and a preprocessing time of O(n (c 2 log n)k⁄k!), where c 1 ,c 2 > 1 are constants. The deterministic preprocessing assumes a weakly nonuniform RAM model; this assumption is not needed if randomization is used in the preprocessing.
---
paper_title: Set similarity search beyond MinHash

paper_content:

We present an improved data structure for approximate similarity (or"nearest neighbor") search under Braun-Blanquet similarity. For $\mathbf{x}, \mathbf{y} \subseteq \{1, \dots, d\}$, Braun-Blanquet similarity is defined as $B(\mathbf{x}, \mathbf{y}) = |\mathbf{x} \cap \mathbf{y}| / \max(|\mathbf{x}|, |\mathbf{y}|)$. Given a collection $P$ of sets, $(b_1,b_2)$-approximate Braun-Blanquet similarity search asks to preprocess $P$ such that given a query set $\mathbf{q}$ for which there exists $\mathbf{x} \in P$ with $B(\mathbf{q},\mathbf{x}) \geq b_1$, we can efficiently return $\mathbf{x}' \in P$ with $B(\mathbf{q},\mathbf{x}')>b_2$. Historically a different measure, the Jaccard similarity $J(\mathbf{x},\mathbf{y}) = |\mathbf{x} \cap \mathbf{y}|/|\mathbf{x} \cup \mathbf{y}|$, has been more widely studied. We argue that it suffices to study regular data structures in which the size of the query set $\mathbf{q}$ and the sizes of sets in $P$ are fixed. In this setting there is a 1-1 mapping between Jaccard similarity and Braun-Blanquet similarity, so the problems are equivalent. It follows from the seminal paper of Indyk and Motwani (STOC 1998) that the so-called MinHash locality-sensitive hash function can be used to construct a data structure for $(j_1,j_2)$-approximate Jaccard similarity with space $O(n^{1+\rho} + dn)$ and query time $O(dn^\rho)$, where $n = |P|$ and $\rho = \log(1/j_1)/ \log(1/j_2)<1$. In the regular setting our data structure improves $\rho$ in the exponent to $\rho' = \log\left(\tfrac{1 + j_1}{2j_1}\right)/\log\left(\tfrac{1+j_2}{2j_2}\right)$ which is always strictly smaller than $\rho$. The exponent in terms of Braun-Blanquet similarity is $\rho'=\log(1/b_1)/ \log(1/b_2)$. Our main technical idea (continued...)
---
paper_title: In Defense of MinHash Over SimHash

paper_content:

MinHash and SimHash are the two widely adopted Locality Sensitive Hashing (LSH) algorithms for large-scale data processing applications. Deciding which LSH to use for a particular problem at hand is an important question, which has no clear answer in the existing literature. In this study, we provide a theoretical answer (validated by experiments) that MinHash virtually always outperforms SimHash when the data are binary, as common in practice such as search. The collision probability of MinHash is a function of resemblance similarity (R), while the collision probability of SimHash is a function of cosine similarity (S). To provide a common basis for comparison, we evaluate retrieval results in terms of S for both MinHash and SimHash. This evaluation is valid as we can prove that MinHash is a valid LSH with respect to S, by using a general inequality S 2
---
paper_title: Dense Associative Memory is Robust to Adversarial Inputs

paper_content:

Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish...
---
paper_title: Fast Exact Search in Hamming Space With Multi-Index Hashing

paper_content:

There is growing interest in representing image data and feature descriptors using compact binary codes for fast near neighbor search. Although binary codes are motivated by their use as direct indices (addresses) into a hash table, codes longer than 32 bits are not being used as such, as it was thought to be ineffective. We introduce a rigorous way to build multiple hash tables on binary code substrings that enables exact k-nearest neighbor search in Hamming space. The approach is storage efficient and straight-forward to implement. Theoretical analysis shows that the algorithm exhibits sub-linear run-time behavior for uniformly distributed codes. Empirical results show dramatic speedups over a linear scan baseline for datasets of up to one billion codes of 64, 128, or 256 bits.
---
paper_title: Optimal Las Vegas Locality Sensitive Data Structures

paper_content:

We show that approximate similarity (near neighbour) search can be solved in high dimensions with performance matching state of the art (data independent) Locality Sensitive Hashing, but with a guarantee of no false negatives. Specifically we give two data structures for common problems. For $c$-approximate near neighbour in Hamming space we get query time $dn^{1/c+o(1)}$ and space $dn^{1+1/c+o(1)}$ matching that of [Indyk and Motwani, 1998] and answering a long standing open question from [Indyk, 2000a] and [Pagh, 2016] in the affirmative. For $(s_1,s_2)$-approximate Jaccard similarity we get query time $d^2n^{\rho+o(1)}$ and space $d^2n^{1+\rho+o(1)}$, $\rho=\log\frac{1+s_1}{2s_1}\big/\log\frac{1+s_2}{2s_2}$, when sets have equal size, matching the performance of [Pagh and Christiani, 2017]. The algorithms are based on space partitions, as with classic LSH, but we construct these using a combination of brute force, tensoring and splitter functions \`a la [Naor et al., 1995]. We also show two dimensionality reduction lemmas with 1-sided error.
---
paper_title: Recurrent-Neural-Network-Based Boolean Factor Analysis and Its Application to Word Clustering

paper_content:

The objective of this paper is to introduce a neural-network-based algorithm for word clustering as an extension of the neural-network-based Boolean factor analysis algorithm (Frolov , 2007). It is shown that this extended algorithm supports even the more complex model of signals that are supposed to be related to textual documents. It is hypothesized that every topic in textual data is characterized by a set of words which coherently appear in documents dedicated to a given topic. The appearance of each word in a document is coded by the activity of a particular neuron. In accordance with the Hebbian learning rule implemented in the network, sets of coherently appearing words (treated as factors) create tightly connected groups of neurons, hence, revealing them as attractors of the network dynamics. The found factors are eliminated from the network memory by the Hebbian unlearning rule facilitating the search of other factors. Topics related to the found sets of words can be identified based on the words' semantics. To make the method complete, a special technique based on a Bayesian procedure has been developed for the following purposes: first, to provide a complete description of factors in terms of component probability, and second, to enhance the accuracy of classification of signals to determine whether it contains the factor. Since it is assumed that every word may possibly contribute to several topics, the proposed method might be related to the method of fuzzy clustering. In this paper, we show that the results of Boolean factor analysis and fuzzy clustering are not contradictory, but complementary. To demonstrate the capabilities of this attempt, the method is applied to two types of textual data on neural networks in two different languages. The obtained topics and corresponding words are at a good level of agreement despite the fact that identical topics in Russian and English conferences contain different sets of keywords.
---
paper_title: Optimal Hashing-based Time-Space Trade-offs for Approximate Near Neighbors

paper_content:

[See the paper for the full abstract.] We show tight upper and lower bounds for time-space trade-offs for the $c$-Approximate Near Neighbor Search problem. For the $d$-dimensional Euclidean space and $n$-point datasets, we develop a data structure with space $n^{1 + \rho_u + o(1)} + O(dn)$ and query time $n^{\rho_q + o(1)} + d n^{o(1)}$ for every $\rho_u, \rho_q \geq 0$ such that: \begin{equation} c^2 \sqrt{\rho_q} + (c^2 - 1) \sqrt{\rho_u} = \sqrt{2c^2 - 1}. \end{equation} This is the first data structure that achieves sublinear query time and near-linear space for every approximation factor $c>1$, improving upon [Kapralov, PODS 2015]. The data structure is a culmination of a long line of work on the problem for all space regimes; it builds on Spherical Locality-Sensitive Filtering [Becker, Ducas, Gama, Laarhoven, SODA 2016] and data-dependent hashing [Andoni, Indyk, Nguyen, Razenshteyn, SODA 2014] [Andoni, Razenshteyn, STOC 2015]. Our matching lower bounds are of two types: conditional and unconditional. First, we prove tightness of the whole above trade-off in a restricted model of computation, which captures all known hashing-based approaches. We then show unconditional cell-probe lower bounds for one and two probes that match the above trade-off for $\rho_q = 0$, improving upon the best known lower bounds from [Panigrahy, Talwar, Wieder, FOCS 2010]. In particular, this is the first space lower bound (for any static data structure) for two probes which is not polynomially smaller than the one-probe bound. To show the result for two probes, we establish and exploit a connection to locally-decodable codes.
---
paper_title: New directions in nearest neighbor searching with applications to lattice sieving

paper_content:

To solve the approximate nearest neighbor search problem (NNS) on the sphere, we propose a method using locality-sensitive filters (LSF), with the property that nearby vectors have a higher probability of surviving the same filter than vectors which are far apart. We instantiate the filters using spherical caps of height 1 -- α, where a vector survives a filter if it is contained in the corresponding spherical cap, and where ideally each filter has an independent, uniformly random direction. For small α, these filters are very similar to the spherical locality-sensitive hash (LSH) family previously studied by Andoni et al. For larger α bounded away from 0, these filters potentially achieve a superior performance, provided we have access to an efficient oracle for finding relevant filters. Whereas existing LSH schemes are limited by a performance parameter of ρ ≥ 1/(2c2 -- 1) to solve approximate NNS with approximation factor c, with spherical LSF we potentially achieve smaller asymptotic values of ρ, depending on the density of the data set. For sparse data sets where the dimension is super-logarithmic in the size of the data set, we asymptotically obtain ρ = 1/(2c2 -- 1), while for a logarithmic dimensionality with density constant κ we obtain asymptotics of ρ ~ 1/(4κc2). To instantiate the filters and prove the existence of an efficient decoding oracle, we replace the independent filters by filters taken from certain structured random product codes. We show that the additional structure in these concatenation codes allows us to decode efficiently using techniques similar to lattice enumeration, and we can find the relevant filters with low overhead, while at the same time not significantly changing the collision probabilities of the filters. We finally apply spherical LSF to sieving algorithms for solving the shortest vector problem (SVP) on lattices, and show that this leads to a heuristic time complexity for solving SVP in dimension n of (3/2)n/2+o(n) a 20.292n+o(n). This asymptotically improves upon the previous best algorithms for solving SVP which use spherical LSH and cross-polytope LSH and run in time 20.298n+o(n). Experiments with the GaussSieve validate the claimed speedup and show that this method may be practical as well, as the polynomial overhead is small.
---
paper_title: Dense Associative Memory for Pattern Recognition

paper_content:

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.
---

Title: Neural Distributed Autoassociative Memories: A Survey

Section 1: INTRODUCTION
Description 1: This section introduces the primary concepts of Neural Associative Memory (NAM), discusses the characteristics of binary vectors, and outlines the scope of the paper, including a generic scheme of NAMs and a summary of the sections to follow.

Section 2: HOPFIELD NAMs
Description 2: Discusses Hopfield networks and their applications, including learning procedures, network dynamics, storage capacity, and retrieval performance in both dense and sparse vector cases.

Section 3: WILLSHAW NAMs
Description 3: Covers Willshaw networks with sparse vectors, exploring their binary connections, learning rules, various strategies for threshold setting, and their implementation in large-scale networks.

Section 4: POTTS NAMs
Description 4: Introduces Potts networks, including their structure, learning rules, dynamics, and variations like the Willshaw-Potts network. It discusses the performance for organization and retrieval of data.

Section 5: GENERALIZATION IN NAMs
Description 5: Examines the function of generalization in NAMs, including internal structures of neuron assemblies, the emergence of prototypes, and hierarchical classification.

Section 6: NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS
Description 6: Discusses NAMs with connections higher than two neurons and those without any connections, exploring alternative network structures and their implications for memory and retrieval.

Section 7: NAMs WITH A BIPARTITE GRAPH STRUCTURE FOR NONBINARY DATA WITH CONSTRAINTS
Description 7: Covers NAMs structured as bipartite graphs for nonbinary data, detailing their use in storing and retrieving large numbers of vectors under specific constraints and iterative algorithms for learning and retrieval.

Section 8: DISCUSSION
Description 8: Provides a comprehensive discussion on the potential and limitations of NAMs, comparing them with other similarity search methods, discussing real-world applications, and outlining directions for future research.