Investigating the generative dynamics of energy-based neural networks
Lorenzo Tausani Alberto Testolin Marco Zorzi
Generative neural networks can produce data samples according to the
statistical properties of their training distribution. This feature can be used
to test modern computational neuroscience hypotheses suggesting that
spontaneous brain activity is partially supported by top-down generative
processing. A widely studied class of generative models is that of Restricted
Boltzmann Machines (RBMs), which can be used as building blocks for
unsupervised deep learning architectures. In this work, we systematically
explore the generative dynamics of RBMs, characterizing the number of states
visited during top-down sampling and investigating whether the heterogeneity of
visited attractors could be increased by starting the generation process from
biased hidden states. By considering an RBM trained on a classic dataset of
handwritten digits, we show that the capacity to produce diverse data
prototypes can be increased by initiating top-down sampling from chimera
states, which encode high-level visual features of multiple digits. We also
found that the model is not capable of transitioning between all possible digit
states within a single generation trajectory, suggesting that the top-down
dynamics is heavily constrained by the shape of the energy function.