ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection
For Transformer models using doubly-stochastic attention, ASAP offers a practical speed-accuracy trade-off by eliminating online iterations, though it is an incremental improvement over existing Sinkhorn and sliced-transport methods.
ASAP replaces the iterative Sinkhorn scaling in doubly-stochastic attention with a fixed sliced-dual operator at inference, achieving 5.3× speedup while matching accuracy on frozen-layer benchmarks and recovering most teacher performance without retraining.
Doubly-stochastic attention has emerged as a transport-based alternative to row-softmax attention, with recent Transformer variants using it to reduce attention sinks and rank collapse while improving performance. In this family, the standard approach is Sinkhorn scaling, which trains more efficiently but still repeats matrix scaling in every inference forward pass. Sliced-transport attention removes the online iteration, but its soft sorting approximation materializes dense tensors for each slice, requiring substantially more training resources than Sinkhorn attention. We introduce ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection, a train-then-compile method that trains the doubly-stochastic layer with Sinkhorn, then replaces the iterative scaling loop at inference with a fixed sliced-dual operator. It learns a lightweight parametric map from exact one-dimensional Kantorovich potentials to the Sinkhorn query-side dual, then reconstructs the attention plan with a two-sided entropic c-transform. Across language and vision benchmarks, ASAP keeps the cheaper training setup and remains highly competitive with recent baselines. In the main frozen-layer benchmark, ASAP is 5.3 faster than the trained Sinkhorn teacher while matching its accuracy; in downstream replacements, ASAP recovers most of the teacher performance without any retraining.