Towards a Common Implementation of Reinforcement Learning for Multiple Robotic Tasks
This work addresses the practical implementation challenge of reinforcement learning for roboticists, though it appears incremental as it builds on value iteration methods with a novel action selection approach.
The authors tackled the problem of reinforcement learning's strong dependency on task-specific tuning in robotics by developing a core implementation that enables learning multiple tasks with minimal per-task adjustments, showing it can outperform existing algorithms in low-dimensional robotic tasks.
Mobile robots are increasingly being employed for performing complex tasks in dynamic environments. Reinforcement learning (RL) methods are recognized to be promising for specifying such tasks in a relatively simple manner. However, the strong dependency between the learning method and the task to learn is a well-known problem that restricts practical implementations of RL in robotics, often requiring major modifications of parameters and adding other techniques for each particular task. In this paper we present a practical core implementation of RL which enables the learning process for multiple robotic tasks with minimal per-task tuning or none. Based on value iteration methods, this implementation includes a novel approach for action selection, called Q-biased softmax regression (QBIASSR), which avoids poor performance of the learning process when the robot reaches new unexplored states. Our approach takes advantage of the structure of the state space by attending the physical variables involved (e.g., distances to obstacles, X,Y,θ pose, etc.), thus experienced sets of states may favor the decision-making process of unexplored or rarely-explored states. This improvement has a relevant role in reducing the tuning of the algorithm for particular tasks. Experiments with real and simulated robots, performed with the software framework also introduced here, show that our implementation is effectively able to learn different robotic tasks without tuning the learning method. Results also suggest that the combination of true online SARSA(λ) with QBIASSR can outperform the existing RL core algorithms in low-dimensional robotic tasks.