Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions
This work addresses a key limitation in offline RL for safety-critical domains like industrial robotics, offering an incremental improvement over existing methods.
The paper tackles the problem of distributional shift in offline reinforcement learning by introducing MoReBRAC, a model-based framework that uses uncertainty-aware latent synthesis to augment training data, achieving significant performance gains on D4RL Gym-MuJoCo benchmarks, especially in random and suboptimal data regimes.
Offline Reinforcement Learning (ORL) holds immense promise for safety-critical domains like industrial robotics, where real-time environmental interaction is often prohibitive. A primary obstacle in ORL remains the distributional shift between the static dataset and the learned policy, which typically mandates high degrees of conservatism that can restrain potential policy improvements. We present MoReBRAC, a model-based framework that addresses this limitation through Uncertainty-Aware latent synthesis. Instead of relying solely on the fixed data, MoReBRAC utilizes a dual-recurrent world model to synthesize high-fidelity transitions that augment the training manifold. To ensure the reliability of this synthetic data, we implement a hierarchical uncertainty pipeline integrating Variational Autoencoder (VAE) manifold detection, model sensitivity analysis, and Monte Carlo (MC) dropout. This multi-layered filtering process guarantees that only transitions residing within high-confidence regions of the learned dynamics are utilized. Our results on D4RL Gym-MuJoCo benchmarks reveal significant performance gains, particularly in ``random'' and ``suboptimal'' data regimes. We further provide insights into the role of the VAE as a geometric anchor and discuss the distributional trade-offs encountered when learning from near-optimal datasets.