A Thorough Examination of Decoding Methods in the Era of LLMs
This work addresses the challenge of selecting effective decoding methods for LLMs across diverse tasks, though it is incremental as it builds on existing research by extending analysis to general-purpose models.
This paper conducted a comprehensive analysis of decoding methods for large language models, finding that performance is highly task-dependent and influenced by factors like alignment and model size, with some methods requiring extensive hyperparameter tuning for optimal results.
Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.