MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training
This addresses the limitation of dependency on stronger models for complex instruction alignment in LLMs, offering a more accessible method, though it appears incremental as it builds on existing self-alignment approaches.
The paper tackles the problem of improving complex instruction-following in Large Language Models without relying on advanced models like GPT-4, proposing a Multi-granularity Self-Contrastive Training framework that achieves significant improvements on benchmarks, surpassing previous self-alignment methods.
Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods.