On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions
This provides foundational theoretical insights for the ML/AI community, bridging approximation theory between feedforward networks and Transformers.
The paper tackles the theoretical understanding of Transformer expressive power by showing that Transformers can approximate maxout networks with comparable complexity, inheriting universal approximation capabilities from ReLU networks, and quantitatively characterizing expressivity via exponential growth in linear regions with depth.
Transformer networks have achieved remarkable empirical success across a wide range of applications, yet their theoretical expressive power remains insufficiently understood. In this paper, we study the expressive capabilities of Transformer architectures. We first establish an explicit approximation of maxout networks by Transformer networks while preserving comparable model complexity. As a consequence, Transformers inherit the universal approximation capability of ReLU networks under similar complexity constraints. Building on this connection, we develop a framework to analyze the approximation of continuous piecewise linear functions by Transformers and quantitatively characterize their expressivity via the number of linear regions, which grows exponentially with depth. Our analysis establishes a theoretical bridge between approximation theory for standard feedforward neural networks and Transformer architectures. It also yields structural insights into Transformers: self-attention layers implement max-type operations, while feedforward layers realize token-wise affine transformations.