Q-function Decomposition with Intervention Semantics with Factored Action Spaces
This work addresses sample complexity issues in reinforcement learning for domains with factored action spaces, offering incremental improvements over existing methods.
The paper tackles the challenge of large combinatorial action spaces in reinforcement learning by proposing a Q-function decomposition method based on causal effect estimation, which improves sample efficiency in both online continuous control and offline sepsis treatment environments.
Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment.