Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity
This addresses the problem of slow Time-to-First-Token for long-context LLM inference on constrained hardware, representing a strong specific gain but incremental improvement over existing sparsification methods.
The paper tackled the computational bottleneck of the prefill stage in large language model inference for long-context workloads by introducing FastForward, a predictive sparsity framework that accelerates prefill through block-wise, context-aware FFN sparsity, achieving up to 1.45x compute-bound speedup at 50% FFN sparsity with less than 6% accuracy loss on LongBench.
The prefill stage of large language model (LLM) inference is a key computational bottleneck for long-context workloads. At short-to-moderate context lengths (1K--16K tokens), Feed-Forward Networks (FFNs) dominate this cost, accounting for most of the total FLOPs. Existing FFN sparsification methods, designed for autoregressive decoding, fail to exploit the prefill stage's parallelism and often degrade accuracy. To address this, we introduce FastForward, a predictive sparsity framework that accelerates LLM prefill through block-wise, context-aware FFN sparsity. FastForward combines (1) a lightweight expert predictor to select high-importance neurons per block, (2) an error compensation network to correct sparsity-induced errors, and (3) a layer-wise sparsity scheduler to allocate compute based on token-mixing importance. Across LLaMA and Qwen models up to 8B parameters, FastForward delivers up to 1.45$\times$ compute-bound speedup at 50% FFN sparsity with $<$ 6% accuracy loss compared to the dense baseline on LongBench, substantially reducing Time-to-First-Token (TTFT) for efficient, long-context LLM inference on constrained hardware.