The Diffusion-Attention Connection
This work provides a foundational unification of key ML tools, potentially impacting all of ML/AI by offering a new theoretical framework.
The paper tackles the problem of unifying disparate mathematical tools in machine learning by showing that transformers, diffusion-maps, and magnetic Laplacians are different regimes of a single Markov geometry based on pre-softmax query-scores, using a QK 'bidivergence' to connect them through equilibrium, nonequilibrium steady-state, and driven dynamics.
Transformers, diffusion-maps, and magnetic Laplacians are usually treated as separate tools; we show they are all different regimes of a single Markov geometry built from pre-softmax query-scores. We define a QK "bidivergence" whose exponentiated and normalized forms yield attention, diffusion-maps, and magnetic diffusion. And use product of experts and Schrödinger-bridges to connect and organize them into equilibrium, nonequilibrium steady-state, and driven dynamics.