COSBO: Conservative Offline Simulation-Based Policy Optimization
This addresses the challenge of enhancing offline policy learning for scenarios where direct real-world interaction is not possible, though it appears incremental as it builds on existing methods.
The paper tackles the problem of offline reinforcement learning being limited to behaviors in training data and simulation environments suffering from a sim-to-real gap, by proposing a method that combines imperfect simulation with target environment data to train policies, resulting in outperforming state-of-the-art approaches like CQL, MOPO, and COMBO in diverse and challenging scenarios.
Offline reinforcement learning allows training reinforcement learning models on data from live deployments. However, it is limited to choosing the best combination of behaviors present in the training data. In contrast, simulation environments attempting to replicate the live environment can be used instead of the live data, yet this approach is limited by the simulation-to-reality gap, resulting in a bias. In an attempt to get the best of both worlds, we propose a method that combines an imperfect simulation environment with data from the target environment, to train an offline reinforcement learning policy. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches CQL, MOPO, and COMBO, especially in scenarios with diverse and challenging dynamics, and demonstrates robust behavior across a variety of experimental conditions. The results highlight that using simulator-generated data can effectively enhance offline policy learning despite the sim-to-real gap, when direct interaction with the real-world is not possible.