Investigating the generative dynamics of energy-based neural networks
This work addresses the problem of understanding generative processing in neural networks for computational neuroscience, but it is incremental as it builds on existing RBM methods.
The study investigated the generative dynamics of Restricted Boltzmann Machines (RBMs) to understand how they produce diverse data samples, finding that initiating sampling from chimera states increased the heterogeneity of visited attractors, but transitions between all digit states were limited due to energy function constraints.
Generative neural networks can produce data samples according to the statistical properties of their training distribution. This feature can be used to test modern computational neuroscience hypotheses suggesting that spontaneous brain activity is partially supported by top-down generative processing. A widely studied class of generative models is that of Restricted Boltzmann Machines (RBMs), which can be used as building blocks for unsupervised deep learning architectures. In this work, we systematically explore the generative dynamics of RBMs, characterizing the number of states visited during top-down sampling and investigating whether the heterogeneity of visited attractors could be increased by starting the generation process from biased hidden states. By considering an RBM trained on a classic dataset of handwritten digits, we show that the capacity to produce diverse data prototypes can be increased by initiating top-down sampling from chimera states, which encode high-level visual features of multiple digits. We also found that the model is not capable of transitioning between all possible digit states within a single generation trajectory, suggesting that the top-down dynamics is heavily constrained by the shape of the energy function.