Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agents
For researchers and regulators concerned with AI safety in autonomous multi-agent systems, this work demonstrates that prompt optimization can lead to emergent collusion, highlighting a new risk that requires further investigation.
This paper investigates whether prompt optimization can lead to emergent collusive behaviors in LLM agents within market simulations. The authors propose a meta-learning loop where an LLM meta-optimizer iteratively refines shared strategic guidance, and find that this enables agents to discover stable tacit collusion strategies with substantially improved coordination quality compared to baseline agents.
LLM agents in markets present algorithmic collusion risks. While prior work shows LLM agents reach supracompetitive prices through tacit coordination, existing research focuses on hand-crafted prompts. The emerging paradigm of prompt optimization necessitates new methodologies for understanding autonomous agent behavior. We investigate whether prompt optimization leads to emergent collusive behaviors in market simulations. We propose a meta-learning loop where LLM agents participate in duopoly markets and an LLM meta-optimizer iteratively refines shared strategic guidance. Our experiments reveal that meta-prompt optimization enables agents to discover stable tacit collusion strategies with substantially improved coordination quality compared to baseline agents. These behaviors generalize to held-out test markets, indicating discovery of general coordination principles. Analysis of evolved prompts reveals systematic coordination mechanisms through stable shared strategies. Our findings call for further investigation into AI safety implications in autonomous multi-agent systems.