Latent Distance Guided Alignment Training for Large Language Models
This work addresses the problem of expensive alignment training for LLM developers, offering an incremental improvement over existing methods like DPO by reducing annotation dependency.
The paper tackles the high cost of human annotation in aligning large language models with human preferences by introducing Latent Distance Guided Alignment Training (LD-Align), which uses latent space distances from sample reconstruction to guide alignment without external annotation, achieving notable alignment in experiments.
Ensuring alignment with human preferences is a crucial characteristic of large language models (LLMs). Presently, the primary alignment methods, RLHF and DPO, require extensive human annotation, which is expensive despite their efficacy. The significant expenses associated with current alignment techniques motivate researchers to investigate the development of annotation-free alignment training methods. In pursuit of improved alignment without relying on external annotation, we introduce Latent Distance Guided Alignment Training (LD-Align). This approach seeks to align the model with a high-quality supervised fine-tune dataset using guidance from a latent space. The latent space is generated through sample reconstruction, akin to auto-encoding. Consequently, we utilize the distance between sample pairs in the latent space to guide DPO-based alignment training. Extensive experimentation and evaluation show the efficacy of our proposed method in achieving notable alignment.