Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning
This addresses the challenge of reusing source data under different dynamics for offline RL, with incremental improvements in handling localized mismatches.
The paper tackles the problem of off-dynamics offline reinforcement learning by proposing LoDADA, a method that clusters transitions to filter source data based on localized dynamics mismatch, resulting in consistent outperformance over state-of-the-art methods.
Off-dynamics offline reinforcement learning (RL) aims to learn a policy for a target domain using limited target data and abundant source data collected under different transition dynamics. Existing methods typically address dynamics mismatch either globally over the state space or via pointwise data filtering; these approaches can miss localized cross-domain similarities or incur high computational cost. We propose Localized Dynamics-Aware Domain Adaptation (LoDADA), which exploits localized dynamics mismatch to better reuse source data. LoDADA clusters transitions from source and target datasets and estimates cluster-level dynamics discrepancy via domain discrimination. Source transitions from clusters with small discrepancy are retained, while those from clusters with large discrepancy are filtered out. This yields a fine-grained and scalable data selection strategy that avoids overly coarse global assumptions and expensive per-sample filtering. We provide theoretical insights and extensive experiments across environments with diverse global and local dynamics shifts. Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch.