Bob and Alice Go to a Bar: Reasoning About Future With Probabilistic Programs
This work offers a foundational shift in reinforcement learning theory by eliminating the need for rewards and exponential trajectory weighting, potentially benefiting researchers in AI and probabilistic programming.
The paper tackles the problem of reinforcement learning by reformulating it as Bayesian inference without using rewards, instead deriving rewards from stochastic agent preferences. It demonstrates the approach on case studies, showing that both multi-agent and single-agent scenarios can be modeled under the same principles.
It is well known that reinforcement learning can be cast as inference in an appropriate probabilistic model. However, this commonly involves introducing a distribution over agent trajectories with probabilities proportional to exponentiated rewards. In this work, we formulate reinforcement learning as Bayesian inference without resorting to rewards, and show that rewards are derived from agent's preferences, rather than the other way around. We argue that agent preferences should be specified stochastically rather than deterministically. Reinforcement learning via inference with stochastic preferences naturally describes agent behaviors, does not require introducing rewards and exponential weighing of trajectories, and allows to reason about agents using the solid foundation of Bayesian statistics. Stochastic conditioning, a probabilistic programming paradigm for conditioning models on distributions rather than values, is the formalism behind agents with probabilistic preferences. We demonstrate realization of our approach on case studies using both a two-agent coordinate game and a single agent acting in a noisy environment, showing that despite superficial differences, both cases can be modeled and reasoned about based on the same principles.