C-MOP: Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution
This work addresses a key bottleneck in prompt optimization for LLMs, offering a novel method to improve performance, though it appears incremental as it builds on existing optimization techniques.
The paper tackles the problem of noisy and conflicting update signals in automatic prompt optimization for Large Language Models (LLMs) by proposing C-MOP, which integrates momentum and boundary-aware clustering to stabilize optimization, resulting in average gains of 1.58% and 3.35% over SOTA baselines and enabling a 3B LLM to surpass a 70B domain-specific model.
Automatic prompt optimization is a promising direction to boost the performance of Large Language Models (LLMs). However, existing methods often suffer from noisy and conflicting update signals. In this research, we propose C-MOP (Cluster-based Momentum Optimized Prompting), a framework that stabilizes optimization via Boundary-Aware Contrastive Sampling (BACS) and Momentum-Guided Semantic Clustering (MGSC). Specifically, BACS utilizes batch-level information to mine tripartite features--Hard Negatives, Anchors, and Boundary Pairs--to precisely characterize the typical representation and decision boundaries of positive and negative prompt samples. To resolve semantic conflicts, MGSC introduces a textual momentum mechanism with temporal decay that distills persistent consensus from fluctuating gradients across iterations. Extensive experiments demonstrate that C-MOP consistently outperforms SOTA baselines like PromptWizard and ProTeGi, yielding average gains of 1.58% and 3.35%. Notably, C-MOP enables a general LLM with 3B activated parameters to surpass a 70B domain-specific dense LLM, highlighting its effectiveness in driving precise prompt evolution. The code is available at https://github.com/huawei-noah/noah-research/tree/master/C-MOP.