Emergent representations in networks trained with the Forward-Forward algorithm
This addresses the problem of biological realism in neural network training for researchers in computational neuroscience and AI, though it appears incremental as it builds on a recently introduced algorithm.
The paper investigates the Forward-Forward algorithm as a biologically plausible alternative to Backpropagation, finding that it produces internal representations with high sparsity and category-specific ensembles, similar to cortical sensory areas.
The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm.