Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
This work addresses sample inefficiency and generalization challenges in continuous control for robotics or autonomous systems, though it is incremental by building on hierarchical reinforcement learning with expert abstractions.
The paper tackles the problem of inefficient decision-making in continuous multi-task environments by using expert-provided abstractions to guide hierarchical reinforcement learning, resulting in improved sample efficiency, task completion rates, and zero-shot generalization compared to existing methods.
Decision-making in complex, continuous multi-task environments is often hindered by the difficulty of obtaining accurate models for planning and the inefficiency of learning purely from trial and error. While precise environment dynamics may be hard to specify, human experts can often provide high-fidelity abstractions that capture the essential high-level structure of a task and user preferences in the target environment. Existing hierarchical approaches often target discrete settings and do not generalize across tasks. We propose a hierarchical reinforcement learning approach that addresses these limitations by dynamically planning over the expert-specified abstraction to generate subgoals to learn a goal-conditioned policy. To overcome the challenges of learning under sparse rewards, we shape the reward based on the optimal state value in the abstract model. This structured decision-making process enhances sample efficiency and facilitates zero-shot generalization. Our empirical evaluation on a suite of procedurally generated continuous control environments demonstrates that our approach outperforms existing hierarchical reinforcement learning methods in terms of sample efficiency, task completion rate, scalability to complex tasks, and generalization to novel scenarios.