Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings
This addresses the challenge of enabling AI systems to generalize to unseen combinations of factors, which is incremental but important for applications like molecular prediction.
The paper tackles the problem of compositional generalization in deep neural networks by using iterated learning with simplicial embeddings, resulting in improved performance over other approaches on vision and molecular graph prediction tasks.
Compositional generalization, the ability of an agent to generalize to unseen combinations of latent factors, is easy for humans but hard for deep neural networks. A line of research in cognitive science has hypothesized a process, ``iterated learning,'' to help explain how human language developed this ability; the theory rests on simultaneous pressures towards compressibility (when an ignorant agent learns from an informed one) and expressivity (when it uses the representation for downstream tasks). Inspired by this process, we propose to improve the compositional generalization of deep networks by using iterated learning on models with simplicial embeddings, which can approximately discretize representations. This approach is further motivated by an analysis of compositionality based on Kolmogorov complexity. We show that this combination of changes improves compositional generalization over other approaches, demonstrating these improvements both on vision tasks with well-understood latent factors and on real molecular graph prediction tasks where the latent structure is unknown.