Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
This work addresses the challenge of causal representation learning in AI, offering a novel approach that could enhance agent autonomy and efficiency, though it appears incremental in building upon existing RL and curiosity-driven methods.
The paper tackles the problem of enabling reinforcement learning agents to discover causal factors in their environment through self-supervised experiments, resulting in agents learning semantically meaningful behaviors with approximately 2.5 times less data than conventional supervised methods and aiding zero-shot learning for complex tasks.
Animals exhibit an innate ability to learn regularities of the world through interaction. By performing experiments in their environment, they are able to discern the causal factors of variation and infer how they affect the world's dynamics. Inspired by this, we attempt to equip reinforcement learning agents with the ability to perform experiments that facilitate a categorization of the rolled-out trajectories, and to subsequently infer the causal factors of the environment in a hierarchical manner. We introduce {\em causal curiosity}, a novel intrinsic reward, and show that it allows our agents to learn optimal sequences of actions and discover causal factors in the dynamics of the environment. The learned behavior allows the agents to infer a binary quantized representation for the ground-truth causal factors in every environment. Additionally, we find that these experimental behaviors are semantically meaningful (e.g., our agents learn to lift blocks to categorize them by weight), and are learnt in a self-supervised manner with approximately 2.5 times less data than conventional supervised planners. We show that these behaviors can be re-purposed and fine-tuned (e.g., from lifting to pushing or other downstream tasks). Finally, we show that the knowledge of causal factor representations aids zero-shot learning for more complex tasks. Visit https://sites.google.com/usc.edu/causal-curiosity/home for website.