Artificial Kuramoto Oscillatory Neurons
This work addresses the need for dynamic representations in neural networks, which is a foundational issue in AI and neuroscience, though it appears incremental as it builds on existing ideas of binding and dynamics.
The authors tackled the problem of static neural representations by introducing Artificial Kuramoto Oscillatory Neurons (AKOrN) as a dynamical alternative to threshold units, resulting in performance improvements across tasks like unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning.
It has long been known in both neuroscience and AI that ``binding'' between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role in both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons (AKOrN) as a dynamical alternative to threshold units, which can be combined with arbitrary connectivity designs such as fully connected, convolutional, or attentive mechanisms. Our generalized Kuramoto updates bind neurons together through their synchronization dynamics. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. We believe that these empirical results show the importance of rethinking our assumptions at the most basic neuronal level of neural representation, and in particular show the importance of dynamical representations. Code:https://github.com/autonomousvision/akorn Project page:https://takerum.github.io/akorn_project_page/