Aligning AI Agents via Information-Directed Sampling
This addresses the challenge of aligning AI agents with human interests in scenarios with unknown preferences and exploration costs, though it is incremental as it builds on existing bandit frameworks.
The paper tackles the AI alignment problem by defining a bandit alignment framework where an agent must maximize long-term reward by interacting with an environment and human with unknown preferences, balancing exploration and exploitation. They show that information-directed sampling achieves favorable regret, outperforming naive methods and Thompson sampling in a toy problem.
The staggering feats of AI systems have brought to attention the topic of AI Alignment: aligning a "superintelligent" AI agent's actions with humanity's interests. Many existing frameworks/algorithms in alignment study the problem on a myopic horizon or study learning from human feedback in isolation, relying on the contrived assumption that the agent has already perfectly identified the environment. As a starting point to address these limitations, we define a class of bandit alignment problems as an extension of classic multi-armed bandit problems. A bandit alignment problem involves an agent tasked with maximizing long-run expected reward by interacting with an environment and a human, both involving details/preferences initially unknown to the agent. The reward of actions in the environment depends on both observed outcomes and human preferences. Furthermore, costs are associated with querying the human to learn preferences. Therefore, an effective agent ought to intelligently trade-off exploration (of the environment and human) and exploitation. We study these trade-offs theoretically and empirically in a toy bandit alignment problem which resembles the beta-Bernoulli bandit. We demonstrate while naive exploration algorithms which reflect current practices and even touted algorithms such as Thompson sampling both fail to provide acceptable solutions to this problem, information-directed sampling achieves favorable regret.