Fast Large Language Model Collaborative Decoding via Speculation
This addresses efficiency issues for users of LLM collaborative decoding, though it is incremental as it builds on existing speculative decoding techniques.
The paper tackles the high computational cost of collaborative decoding in large language models by introducing Collaborative decoding via Speculation (CoS), a framework that accelerates the process without compromising performance, achieving speedups of 1.11x to 2.23x in experiments.
Large Language Model (LLM) collaborative decoding techniques improve output quality by combining the outputs of multiple models at each generation step, but they incur high computational costs. In this paper, we introduce Collaborative decoding via Speculation (CoS), a novel framework that accelerates collaborative decoding without compromising performance. Inspired by Speculative Decoding--where a small proposal model generates tokens sequentially, and a larger target model verifies them in parallel, our approach builds on two key insights: (1) the verification distribution can be the combined distribution of both the proposal and target models, and (2) alternating each model as the proposer and verifier can further enhance efficiency. We generalize this method to collaboration among n models and theoretically prove that CoS is never slower than standard collaborative decoding, typically achieving faster speed. Extensive experiments demonstrate CoS is 1.11x-2.23x faster than standard collaborative decoding without compromising generation quality. Our code is available at https://github.com/Kamichanw/CoS/.