Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
This addresses a practical issue in robot control using reinforcement learning, where early termination for safety can cause learning failures due to reward design flaws, offering an incremental improvement to exception handling.
The paper tackles the problem of unintentional overestimation in temporal-difference learning when episodes terminate early, which can lead to wrong policies, by proposing a method to intentionally underestimate the value at termination and adjusting it based on stationarity. Simulations and real robot experiments demonstrated that the method stably obtains optimal policies across various tasks and reward designs.
Robot control using reinforcement learning has become popular, but its learning process generally terminates halfway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentionally implicit underestimation or overestimation occurs, depending on the reward design in the normal states. When the episode is terminated due to task failure, the failure may be highly valued with the unintentional overestimation, and the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in practical use of TD learning to review the exception handling at termination. This paper therefore proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. In addition, the degree of underestimation is adjusted according to the degree of stationarity at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulations and real robot experiments showed that the proposed method can stably obtain the optimal policies for various tasks and reward designs. https://youtu.be/AxXr8uFOe7M