Cross apprenticeship learning framework: Properties and solution approaches
This work addresses apprenticeship learning for scenarios where experts operate in varied environments, offering a principled approach to policy generalization, though it appears incremental as an extension of existing apprenticeship learning methods.
The paper tackles the problem of learning policies from expert trajectories across multiple environments with different dynamics, introducing the cross apprenticeship learning (CAL) framework to balance environment-specific performance and cross-environment consistency. It demonstrates the framework's properties and a convex approximation in a windy gridworld navigation task.
Apprenticeship learning is a framework in which an agent learns a policy to perform a given task in an environment using example trajectories provided by an expert. In the real world, one might have access to expert trajectories in different environments where the system dynamics is different while the learning task is the same. For such scenarios, two types of learning objectives can be defined. One where the learned policy performs very well in one specific environment and another when it performs well across all environments. To balance these two objectives in a principled way, our work presents the cross apprenticeship learning (CAL) framework. This consists of an optimization problem where an optimal policy for each environment is sought while ensuring that all policies remain close to each other. This nearness is facilitated by one tuning parameter in the optimization problem. We derive properties of the optimizers of the problem as the tuning parameter varies. Since the problem is nonconvex, we provide a convex outer approximation. Finally, we demonstrate the attributes of our framework in the context of a navigation task in a windy gridworld environment.