From Imitation to Refinement -- Residual RL for Precise Assembly
This addresses the challenge of achieving precise and reliable robotic manipulation for tasks like assembly, though it appears incremental as it builds on existing BC and RL techniques.
The paper tackles the problem of unreliable performance saturation in Behavior Cloning for precise robotic assembly tasks by proposing ResiP, a method that augments a frozen BC model with a closed-loop residual RL policy, resulting in improved reliability while retaining ease of teaching.
Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks. However, we find that the ease of teaching comes at the cost of unreliable performance that saturates with increasing data for tasks requiring precision. The performance saturation can be attributed to two critical factors: (a) distribution shift resulting from the use of offline data and (b) the lack of closed-loop corrective control caused by action chucking (predicting a set of future actions executed open-loop) critical for BC performance. Our key insight is that by predicting action chunks, BC policies function more like trajectory "planners" than closed-loop controllers necessary for reliable execution. To address these challenges, we devise a simple yet effective method, ResiP (Residual for Precise Manipulation), that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities. ResiP augments a frozen, chunked BC model with a fully closed-loop residual policy trained with reinforcement learning (RL) that addresses distribution shifts and introduces closed-loop corrections over open-loop execution of action chunks predicted by the BC trajectory planner. Videos, code, and data: https://residual-assembly.github.io.