Simultaneous active parameter estimation and control using sampling-based Bayesian reinforcement learning
For robotic manipulation, it provides a method to trade off exploration and exploitation under parameter uncertainty, outperforming certainty equivalent MPC in simulations.
This work frames simultaneous parameter estimation and control as a Bayes-adaptive MDP, solved online with MCTS and an EKF, achieving near-optimal goal-reaching while reducing model uncertainty in continuous spaces with Gaussian noise.
Robots performing manipulation tasks must operate under uncertainty about both their pose and the dynamics of the system. In order to remain robust to modeling error and shifts in payload dynamics, agents must simultaneously perform estimation and control tasks. However, the optimal estimation actions are often not the optimal actions for accomplishing the control tasks, and thus agents trade between exploration and exploitation. This work frames the problem as a Bayes-adaptive Markov decision process and solves it online using Monte Carlo tree search and an extended Kalman filter to handle Gaussian process noise and parameter uncertainty in a continuous space. MCTS selects control actions to reduce model uncertainty and reach the goal state nearly optimally. Certainty equivalent model predictive control is used as a benchmark to compare performance in simulations with varying process noise and parameter uncertainty.