Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO
This addresses security vulnerabilities in decentralized LLM training, which is an incremental but important step for practical deployment.
The paper tackles adversarial attacks in decentralized Group Relative Policy Optimization (GRPO) for LLM post-training, showing that malicious tokens can poison benign nodes with up to 100% success in 50 iterations, and proposes defenses achieving up to 100% stop rates.
Group Relative Policy Optimization (GRPO) has demonstrated great utilization in post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and, through reinforcement learning, preferred completions are learnt. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and then exchanged in the forms of strings. In this work, we present the first adversarial attack in decentralised GRPO. We demonstrate that malicious parties can poison such systems by injecting arbitrary malicious tokens in benign models in both out-of-context and in-context attacks. Using empirical examples of math and coding tasks, we show that adversarial attacks can easily poison the benign nodes, polluting their local LLM post-training, achieving attack success rates up to 100% in as few as 50 iterations. We propose two ways to defend against these attacks, depending on whether all users train the same model or different models. We show that these defenses can achieve stop rates of up to 100%, making the attack impossible.