Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functions
This work addresses performance issues in RCPs for MABs, offering an incremental improvement to make them competitive with established methods.
The paper tackled the problem of reward-conditioned policies (RCPs) underperforming in multi-armed bandit (MAB) problems by introducing normalized weight functions for generalized marginalization, resulting in improved convergence and competitive performance with classic methods, especially in large action spaces and sparse reward scenarios.
Recently proposed reward-conditioned policies (RCPs) offer an appealing alternative in reinforcement learning. Compared with policy gradient methods, policy learning in RCPs is simpler since it is based on supervised learning, and unlike value-based methods, it does not require optimization in the action space to take actions. However, for multi-armed bandit (MAB) problems, we find that RCPs are slower to converge and have inferior expected rewards at convergence, compared with classic methods such as the upper confidence bound and Thompson sampling. In this work, we show that the performance of RCPs can be enhanced by constructing policies through the marginalization of rewards using normalized weight functions, whose sum or integral equal $1$, although the function values may be negative. We refer to this technique as generalized marginalization, whose advantage is that negative weights for policies conditioned on low rewards can make the resulting policies more distinct from them. Strategies to perform generalized marginalization in MAB with discrete action spaces are studied. Through simulations, we demonstrate that the proposed technique improves RCPs and makes them competitive with classic methods, showing superior performance on challenging MABs with large action spaces and sparse reward signals.