Multi-Amateur Contrastive Decoding for Text Generation
This work addresses the problem of enhancing text generation quality for users of language models by offering a more robust inference-time method, though it is incremental as it builds on existing contrastive decoding frameworks.
The paper tackled the limitation of Contrastive Decoding in text generation by proposing Multi-Amateur Contrastive Decoding (MACD), which uses an ensemble of amateur models to better capture diverse failure modes like repetition and hallucination, resulting in improved fluency, coherence, and diversity across multiple domains without extra training.
Contrastive Decoding (CD) has emerged as an effective inference-time strategy for enhancing open-ended text generation by exploiting the divergence in output probabilities between a large expert language model and a smaller amateur model. Although CD improves coherence and fluency, its dependence on a single amateur restricts its capacity to capture the diverse and multifaceted failure modes of language generation, such as repetition, hallucination, and stylistic drift. This paper proposes Multi-Amateur Contrastive Decoding (MACD), a generalization of the CD framework that employs an ensemble of amateur models to more comprehensively characterize undesirable generation patterns. MACD integrates contrastive signals through both averaging and consensus penalization mechanisms and extends the plausibility constraint to operate effectively in the multi-amateur setting. Furthermore, the framework enables controllable generation by incorporating amateurs with targeted stylistic or content biases. Experimental results across multiple domains, such as news, encyclopedic, and narrative, demonstrate that MACD consistently surpasses conventional decoding methods and the original CD approach in terms of fluency, coherence, diversity, and adaptability, all without requiring additional training or fine-tuning.