Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
This addresses the problem of handling complex, multi-modal tasks in robotics or AI by enabling hierarchical policy learning from raw demonstrations, though it is incremental as it builds on existing frameworks like GAIL and Options.
The paper tackles the challenge of learning hierarchical policies from unsegmented demonstrations in imitation learning by proposing Directed-Info GAIL, which uses a directed graphical model to discover sub-task interactions and maximize directed information flow, resulting in automatic learning of sub-task policies without prior segmentation.
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.