Accumulating Risk Capital Through Investing in Cooperation
This work addresses safety and cooperation trade-offs in multi-agent systems, which is incremental as it builds on existing methods.
The paper tackles the trade-off between promoting cooperation and vulnerability to exploitation in multi-agent learning, proposing an objective that balances safety and long-term cooperation, with results showing exponentially large returns from small risks in iterated games.
Recent work on promoting cooperation in multi-agent learning has resulted in many methods which successfully promote cooperation at the cost of becoming more vulnerable to exploitation by malicious actors. We show that this is an unavoidable trade-off and propose an objective which balances these concerns, promoting both safety and long-term cooperation. Moreover, the trade-off between safety and cooperation is not severe, and you can receive exponentially large returns through cooperation from a small amount of risk. We study both an exact solution method and propose a method for training policies that targets this objective, Accumulating Risk Capital Through Investing in Cooperation (ARCTIC), and evaluate them in iterated Prisoner's Dilemma and Stag Hunt.