Infomax strategies for an optimal balance between exploration and exploitation
This addresses decision-making in uncertain environments, such as reinforcement learning, but is incremental as it builds on existing Infomax principles.
The paper tackles the multi-armed bandit problem by proposing an Infomax strategy (Info-p) that optimally balances exploration and exploitation, showing it saturates known optimal bounds and compares favorably to existing policies.
Proper balance between exploitation and exploration is what makes good decisions, which achieve high rewards like payoff or evolutionary fitness. The Infomax principle postulates that maximization of information directs the function of diverse systems, from living systems to artificial neural networks. While specific applications are successful, the validity of information as a proxy for reward remains unclear. Here, we consider the multi-armed bandit decision problem, which features arms (slot-machines) of unknown probabilities of success and a player trying to maximize cumulative payoff by choosing the sequence of arms to play. We show that an Infomax strategy (Info-p) which optimally gathers information on the highest mean reward among the arms saturates known optimal bounds and compares favorably to existing policies. The highest mean reward considered by Info-p is not the quantity actually needed for the choice of the arm to play, yet it allows for optimal tradeoffs between exploration and exploitation.