Dispatch-Aware Ragged Attention for Pruned Vision Transformers
For ViT practitioners using token pruning, this work addresses the overlooked dispatch bottleneck that prevents pruning FLOP reductions from translating to speedups.
Token pruning in ViTs reduces attention FLOPs but not wall-clock latency due to dispatch overhead (60-90 μs). A bidirectional Triton kernel cuts dispatch to ~40 μs, achieving up to 2.24x end-to-end throughput over padded SDPA across four pruning algorithms and three DeiT variants with negligible accuracy loss.
Token pruning methods for Vision Transformers (ViTs) promise quadratic reductions in attention FLOPs by dropping uninformative patches. Yet when pruned sequences are executed with state-of-the-art variable-length attention APIs -- including FlashAttention-2's varlen and PyTorch's NestedTensor SDPA-the wall-clock attention latency doesn't scale accordingly. We trace this to a dispatch-overhead bottleneck: at the short, post-pruning sequence lengths typical of ViTs (<=197 tokens), actual matrix arithmetic completes in single-digit microseconds while the host-side dispatch path consumes 60-90 us. We present a lightweight, bidirectional Triton attention kernel whose dispatch floor is 40 us roughly 1.5x lower than FlashAttention-2 varlen-allowing pruning savings to become more visible in wall-clock time. Integrated into a complete pack-attend-unpack pipeline, our system achieves up to 2.24x end-to-end throughput over padded PyTorch SDPA consistently across four pruning algorithms (Threshold-L2, DynamicViT, EViT, ATS), scales across DeiT-T/S/B, and maintains bit-exact classification predictions with <0.007 max absolute logit difference.