Exploration Unbound
This work addresses a fundamental shift in exploration-exploitation trade-offs for decision-making agents in complex environments, potentially impacting reinforcement learning and AI systems.
The paper tackles the problem of sequential decision-making in environments where exploration remains beneficial indefinitely, presenting a simple example where optimal agents maintain a propensity to explore forever due to unbounded rewards and continuous learning opportunities.
A sequential decision-making agent balances between exploring to gain new knowledge about an environment and exploiting current knowledge to maximize immediate reward. For environments studied in the traditional literature, optimal decisions gravitate over time toward exploitation as the agent accumulates sufficient knowledge and the benefits of further exploration vanish. What if, however, the environment offers an unlimited amount of useful knowledge and there is large benefit to further exploration no matter how much the agent has learned? We offer a simple, quintessential example of such a complex environment. In this environment, rewards are unbounded and an agent can always increase the rate at which rewards accumulate by exploring to learn more. Consequently, an optimal agent forever maintains a propensity to explore.