Discrete, compositional, and symbolic representations through attractor
  dynamics
Andrew Nam Eric Elmoznino Nikolay Malkin Chen Sun Yoshua Bengio Guillaume Lajoie
Compositionality is an important feature of discrete symbolic systems, such
as language and programs, as it enables them to have infinite capacity despite
a finite symbol set. It serves as a useful abstraction for reasoning in both
cognitive science and in AI, yet the interface between continuous and symbolic
processing is often imposed by fiat at the algorithmic level, such as by means
of quantization or a softmax sampling step. In this work, we explore how
discretization could be implemented in a more neurally plausible manner through
the modeling of attractor dynamics that partition the continuous representation
space into basins that correspond to sequences of symbols. Building on
established work in attractor networks and introducing novel training methods,
we show that imposing structure in the symbolic space can produce
compositionality in the attractor-supported representation space of rich
sensory inputs. Lastly, we argue that our model exhibits the process of an
information bottleneck that is thought to play a role in conscious experience,
decomposing the rich information of a sensory input into stable components
encoding symbolic information.