Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data
For computational neuroscience and biologically plausible deep learning, this work identifies which local learning rules can match backpropagation's performance on hierarchical tasks, revealing a key limitation of direct feedback approaches.
The study investigates biologically plausible local learning rules on the Random Hierarchy Model (RHM) and finds that direct feedback-based rules fail due to input-specific nonlinearities, while layerwise self-supervised contrastive/non-contrastive rules successfully learn hierarchical structure with data efficiency matching supervised backpropagation.
The brain learns abstract representations of high-dimensional sensory input, but the plasticity rules that enable such learning are unknown. We study biologically plausible algorithms on the Random Hierarchy Model (RHM), an artificial dataset designed to investigate how deep neural networks learn the intrinsic hierarchical structure of high-dimensional data. We focus on two types of local learning rules that avoid both a long convergence time and the use of a symmetric error network. The first type uses direct feedback signals to approximate error propagation from the output layer. The second type uses layerwise self-supervised contrastive or non-contrastive loss functions that do not explicitly approximate errors at the output layer. We show that all rules of the first type fail to solve the tasks of the RHM and trace this failure back to input-specific nonlinearities (`masking') that are implemented in full backpropagation and are essential for learning complex tasks. However, algorithms of the second type are able to learn the hierarchical hidden structure of the RHM tasks and are as data-efficient as supervised backpropagation training, while being compatible with known rules of synaptic plasticity in cortex.