Measure-to-measure interpolation using Transformers
This provides a theoretical foundation for Transformers' capabilities in handling diverse data types, though it is incremental as it builds on existing transport map assumptions.
The paper tackles the problem of understanding Transformers' expressive power as measure-to-measure maps, showing that a single Transformer can be parameterized to exactly match N arbitrary input measures to N arbitrary target measures, given the existence of transport maps between each pair.
Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-point maps, a Transformer acts as a measure-to-measure map implemented as specific interacting particle system on the unit sphere: the input is the empirical measure of tokens in a prompt and its evolution is governed by the continuity equation. In fact, Transformers are not limited to empirical measures and can in principle process any input measure. As the nature of data processed by Transformers is expanding rapidly, it is important to investigate their expressive power as maps from an arbitrary measure to another arbitrary measure. To that end, we provide an explicit choice of parameters that allows a single Transformer to match $N$ arbitrary input measures to $N$ arbitrary target measures, under the minimal assumption that every pair of input-target measures can be matched by some transport map.