MO2: Model-Based Offline Options
This addresses the challenge of sample-efficient bottleneck option discovery for embodied intelligence, enabling transfer to new tasks, though it appears incremental by extending prior work to offline and continuous domains.
The paper tackles the problem of discovering useful behaviors from past experience for transfer to new tasks, particularly in continuous state-action spaces, and introduces Model-Based Offline Options (MO2), which leads to performance exceeding recent option learning methods on complex long-horizon continuous control tasks with sparse, delayed rewards.
The ability to discover useful behaviours from past experience and transfer them to new tasks is considered a core component of natural embodied intelligence. Inspired by neuroscience, discovering behaviours that switch at bottleneck states have been long sought after for inducing plans of minimum description length across tasks. Prior approaches have either only supported online, on-policy, bottleneck state discovery, limiting sample-efficiency, or discrete state-action domains, restricting applicability. To address this, we introduce Model-Based Offline Options (MO2), an offline hindsight framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces. Once bottleneck options are learnt offline over source domains, they are transferred online to improve exploration and value estimation on the transfer domain. Our experiments show that on complex long-horizon continuous control tasks with sparse, delayed rewards, MO2's properties are essential and lead to performance exceeding recent option learning methods. Additional ablations further demonstrate the impact on option predictability and credit assignment.