Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
This addresses the challenge of developing task-agnostic, unsupervised altruistic agents for multi-agent systems, which is incremental as it builds on existing reinforcement learning methods but introduces a novel formalization for learning without external supervision.
The paper tackles the problem of enabling reinforcement learning agents to learn altruistic behaviors without external rewards or explicit knowledge of other agents' goals, by formalizing an approach where agents maximize the number of reachable states for others. The result shows that these unsupervised agents perform comparably to or even outperform explicitly trained cooperative agents in multi-agent environments.
Can artificial agents learn to assist others in achieving their goals without knowing what those goals are? Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for benefiting other agents in a given situation. Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals. However, explicit knowledge of other agents' goals is often difficult to acquire. In the case of human agents, their goals and preferences may be difficult to express fully; they might be ambiguous or even contradictory. Thus, it is beneficial to develop agents that do not depend on external supervision and learn altruistic behaviour in a task-agnostic manner. We propose to act altruistically towards other agents by giving them more choice and allowing them to achieve their goals better. Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference. We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future. We evaluate our approach in three different multi-agent environments where another agent's success depends on altruistic behaviour. Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.