Transformer Approximations from ReLUs
Provides new analytical tools for analyzing softmax transformer models, but the results are theoretical and incremental.
The paper provides a systematic recipe for translating ReLU approximation results to softmax attention, yielding target-specific economic resource bounds. It demonstrates the recipe on multiplication, reciprocal, and min/max primitives.
We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.