Physical Derivatives: Computing policy gradients by physical forward-propagation
This addresses the trade-off between model-free and model-based RL for robotics, offering a middle ground to reduce learning costs without full model bias, though it appears incremental.
The paper tackles the problem of reducing the cost of policy learning in reinforcement learning by learning the sensitivity of trajectories to parameter perturbations instead of a full dynamic model, and demonstrates feasibility on a custom-built physical robot.
Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.