The Partially Observable History Process
This provides a streamlined framework for algorithm design and theory development in reinforcement learning, applicable across single and multi-agent domains, though it appears incremental as it builds on existing models.
The paper tackles the problem of unifying single and multi-agent reinforcement learning models by introducing the partially observable history process (POHP) formalism, which abstracts away other players without reducing them to stochastic processes and unifies traditional models like Markov decision processes and Markov games without adding technical complexity.
We introduce the partially observable history process (POHP) formalism for reinforcement learning. POHP centers around the actions and observations of a single agent and abstracts away the presence of other players without reducing them to stochastic processes. Our formalism provides a streamlined interface for designing algorithms that defy categorization as exclusively single or multi-agent, and for developing theory that applies across these domains. We show how the POHP formalism unifies traditional models including the Markov decision process, the Markov game, the extensive-form game, and their partially observable extensions, without introducing burdensome technical machinery or violating the philosophical underpinnings of reinforcement learning. We illustrate the utility of our formalism by concisely exploring observable sequential rationality, examining some theoretical properties of general immediate regret minimization, and generalizing the extensive-form regret minimization (EFR) algorithm.