Neural Teleportation
This work provides a new mathematical tool for understanding and manipulating neural network weight spaces, potentially benefiting researchers and practitioners in machine learning by offering novel ways to analyze and optimize network training.
The paper introduces "neural teleportation," a process derived from quiver representation theory that moves a neural network in weight space while preserving its function. This operation can explore loss level curves, alter the local loss landscape, sharpen global minima, and boost back-propagated gradients.
In this paper, we explore a process called neural teleportation, a mathematical consequence of applying quiver representation theory to neural networks. Neural teleportation "teleports" a network to a new position in the weight space and preserves its function. This phenomenon comes directly from the definitions of representation theory applied to neural networks and it turns out to be a very simple operation that has remarkable properties. We shed light on surprising and counter-intuitive consequences neural teleportation has on the loss landscape. In particular, we show that teleportation can be used to explore loss level curves, that it changes the local loss landscape, sharpens global minima and boosts back-propagated gradients at any moment during the learning process. Our results can be reproduced with the code available here: https://github.com/vitalab/neuralteleportation