Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence
This addresses the problem of slow reasoning processes for users of large language models, though it is incremental as it builds on existing parallelizability concepts.
The paper tackles the computational inefficiency of generating lengthy reasoning sequences by introducing a method that decodes multiple tokens per forward pass using a tree-like attention mask within a single sequence, achieving up to nearly 100% speedup in decoding while maintaining answer quality.
Recent advances in reasoning models have demonstrated significant improvements in accuracy by employing detailed and comprehensive reasoning processes. However, generating these lengthy reasoning sequences is computationally expensive and time-consuming. To address this inefficiency, we leverage the inherent parallelizability of certain tasks to accelerate the reasoning process. Specifically, when multiple parallel reasoning steps exist, we decode multiple tokens per forward pass via a tree-like attention mask within a single sequence, avoiding additional memory usage. Experimental results show that our method achieves up to nearly 100\% speedup in decoding while basically maintaining the answer quality.