Approximation Theory for Lipschitz Continuous Transformers
This provides a rigorous theoretical foundation for designing robust Transformers in safety-sensitive settings, addressing a known bottleneck.
The paper tackles the lack of approximation-theoretic guarantees for Lipschitz continuous Transformers by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction, proving a universal approximation theorem within a Lipschitz-constrained function space with guarantees independent of token count.
Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.