Multi-Environment Pretraining Enables Transfer to Action Limited Datasets
This addresses a key bottleneck in reinforcement learning for scenarios like video game-play where action data is scarce, offering a practical solution for transfer learning with limited annotations.
The paper tackled the challenge of training reinforcement learning models when target environment data lacks action annotations by proposing Action Limited PreTraining (ALPT), which uses inverse dynamics modelling to label missing actions by leveraging fully-annotated source environments, resulting in significant improvements in game performance and generalization with only 12 minutes of annotated gameplay data.
Using massive datasets to train large-scale models has emerged as a dominant approach for broad generalization in natural language and vision applications. In reinforcement learning, however, a key challenge is that available data of sequential decision making is often not annotated with actions - for example, videos of game-play are much more available than sequences of frames paired with their logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated datasets from a \emph{target} environment of interest with fully-annotated datasets from various other \emph{source} environments. Our method, Action Limited PreTraining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating action labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that we can significantly improve game performance and generalization capability compared to other approaches, using annotated datasets equivalent to only $12$ minutes of gameplay. Highlighting the power of IDM, we show that these benefits remain even when target and source environments share no common actions.