Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities
This work provides a foundational taxonomy for researchers in reinforcement learning to systematize transfer methods, but it is incremental as it organizes existing concepts rather than introducing new techniques.
The paper tackles the problem of knowledge transfer in reinforcement learning by proposing a taxonomy of knowledge modalities, such as dynamics models and policies, to guide approaches for improving generalization and efficiency, but does not report concrete numerical results.
Contemporary artificial intelligence systems exhibit rapidly growing abilities accompanied by the growth of required resources, expansive datasets and corresponding investments into computing infrastructure. Although earlier successes predominantly focus on constrained settings, recent strides in fundamental research and applications aspire to create increasingly general systems. This evolving landscape presents a dual panorama of opportunities and challenges in refining the generalisation and transfer of knowledge - the extraction from existing sources and adaptation as a comprehensive foundation for tackling new problems. Within the domain of reinforcement learning (RL), the representation of knowledge manifests through various modalities, including dynamics and reward models, value functions, policies, and the original data. This taxonomy systematically targets these modalities and frames its discussion based on their inherent properties and alignment with different objectives and mechanisms for transfer. Where possible, we aim to provide coarse guidance delineating approaches which address requirements such as limiting environment interactions, maximising computational efficiency, and enhancing generalisation across varying axes of change. Finally, we analyse reasons contributing to the prevalence or scarcity of specific forms of transfer, the inherent potential behind pushing these frontiers, and underscore the significance of transitioning from designed to learned transfer.