SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences
This addresses performance degradation in long-sequence tasks for AI/ML practitioners, though it is incremental as it builds on existing sparse attention methods.
The paper tackles the quadratic complexity of self-attention in transformers for long sequences by proposing SALO, an efficient spatial accelerator enabling hybrid sparse attention mechanisms, achieving average speedups of 17.66x and 89.33x compared to GPU and CPU implementations.
The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence length incurs heavy computational and memory burdens, especially for tasks with long sequences. Existing accelerators face performance degradation in these tasks. To this end, we propose SALO to enable hybrid sparse attention mechanisms for long sequences. SALO contains a data scheduler to map hybrid sparse attention patterns onto hardware and a spatial accelerator to perform the efficient attention computation. We show that SALO achieves 17.66x and 89.33x speedup on average compared to GPU and CPU implementations, respectively, on typical workloads, i.e., Longformer and ViL.