Guided Policy Search with Delayed Sensor Measurements
This addresses a practical issue in robotics for tasks with sensor delays, but it is incremental as it adapts an existing method to a specific scenario.
The paper tackled the problem of applying guided policy search to non-Markovian dynamics caused by delayed sensor measurements, specifically in pouring a precise amount of liquid, and showed that simple state augmentation can extend the method to such systems.
Guided policy search is a method for reinforcement learning that trains a general policy for accomplishing a given task by guiding the learning of the policy with multiple guiding distributions. Guided policy search relies on learning an underlying dynamical model of the environment and then, at each iteration of the algorithm, using that model to gradually improve the policy. This model, though, often makes the assumption that the environment dynamics are markovian, e.g., depend only on the current state and control signal. In this paper we apply guided policy search to a problem with non-markovian dynamics. Specifically, we apply it to the problem of pouring a precise amount of liquid from a cup into a bowl, where many of the sensor measurements experience non-trivial amounts of delay. We show that, with relatively simple state augmentation, guided policy search can be extended to non-markovian dynamical systems, where the non-markovianess is caused by delayed sensor readings.