Quinoa: a Q-function You Infer Normalized Over Actions
This work addresses the challenge of policy optimization in reinforcement learning for researchers and practitioners, offering an incremental improvement by removing the need for separate policy optimization steps.
The paper tackled the problem of learning action-value functions in continuous action spaces by introducing a method that combines normalizing flows with value-function learning, resulting in a unified objective that simplifies Actor-Critic algorithms and enables fast, exact sampling of multimodal policies.
We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.