A unified framework for establishing the universal approximation of transformer-type architectures
This work provides a foundational theoretical framework for understanding and designing transformer architectures with guaranteed universal approximation, which is incremental but broadens prior results.
The authors tackled the problem of proving universal approximation for transformer architectures by developing a unified theoretical framework that identifies token distinguishability as a key requirement and provides a general sufficient condition, enabling proofs for various attention mechanisms like kernel-based and sparse ones.
We investigate the universal approximation property (UAP) of transformer-type architectures, providing a unified theoretical framework that extends prior results on residual networks to models incorporating attention mechanisms. Our work identifies token distinguishability as a fundamental requirement for UAP and introduces a general sufficient condition that applies to a broad class of architectures. Leveraging an analyticity assumption on the attention layer, we can significantly simplify the verification of this condition, providing a non-constructive approach in establishing UAP for such architectures. We demonstrate the applicability of our framework by proving UAP for transformers with various attention mechanisms, including kernel-based and sparse attention mechanisms. The corollaries of our results either generalize prior works or establish UAP for architectures not previously covered. Furthermore, our framework offers a principled foundation for designing novel transformer architectures with inherent UAP guarantees, including those with specific functional symmetries. We propose examples to illustrate these insights.