Learning compositional functions via multiplicative weight updates
This addresses the need for robust optimization in neural networks, particularly for applications requiring stable training without manual tuning, though it appears incremental as it adapts existing methods like Adam.
The paper tackles the problem of learning compositional functions in neural networks, which often suffer from vanishing and exploding gradients with gradient descent, by proving that multiplicative weight updates enable a descent lemma and introducing Madam, a multiplicative version of Adam that trains state-of-the-art architectures without learning rate tuning.
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.