Motif: Intrinsic Motivation from Artificial Intelligence Feedback
This addresses the problem of sample-efficient exploration in open-ended environments for AI agents, representing a novel approach rather than an incremental improvement.
The authors tackled the challenge of enabling agents to explore and evaluate actions in rich environments without prior knowledge by proposing Motif, a method that uses LLM preferences over captions to create intrinsic rewards for reinforcement learning. In NetHack, Motif achieved higher scores than direct score maximization and outperformed existing methods when combined with environment rewards, making progress on previously unsolved tasks.
Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging. In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent. Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning. We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game. Surprisingly, by only learning to maximize its intrinsic reward, Motif achieves a higher game score than an algorithm directly trained to maximize the score itself. When combining Motif's intrinsic reward with the environment reward, our method significantly outperforms existing approaches and makes progress on tasks where no advancements have ever been made without demonstrations. Finally, we show that Motif mostly generates intuitive human-aligned behaviors which can be steered easily through prompt modifications, while scaling well with the LLM size and the amount of information given in the prompt.