Speculative Contrastive Decoding
This addresses the problem of high computational costs and exposure bias in LLM inference for users in natural language processing, though it is incremental as it builds on existing speculative and contrastive decoding methods.
The paper tackles the computational inefficiency and sub-optimal quality in auto-regressive inference of large language models by introducing Speculative Contrastive Decoding (SCD), which uses predictions from smaller models to achieve both faster decoding and improved performance, as demonstrated across four diverse language tasks.
Large language models~(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding~(SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models~(LMs) to achieve both decoding acceleration and quality improvement. Extensive evaluations and analyses on four diverse language tasks demonstrate the effectiveness of SCD, showing that decoding efficiency and quality can compatibly benefit from one smaller LM.