Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment
This work addresses the challenge of maintaining conversational abilities in computationally efficient quantized LLMs, which is crucial for deploying effective chatbots, though it is incremental as it builds on existing quantization and alignment methods.
The paper tackles the problem of conversational performance degradation in quantized large language models due to token-flipping, proposing quantization-aware direct preference optimization (QDPO) to align them with full-precision models, resulting in superior performance improvements over existing techniques.
The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.