Generalization in Multimodal Language Learning from Simulation
This work addresses the challenge of systematic reasoning in AI for multimodal learning, though it is incremental as it builds on existing studies of generalization in neural networks.
The study tackled the problem of compositional generalization in neural networks by investigating how multimodal input sequences from simulation affect generalization performance, finding that multimodality strongly improves generalization in settings where vision-only models struggle, with improvements linked to factors like the number of objects and color overlaps.
Neural networks can be powerful function approximators, which are able to model high-dimensional feature distributions from a subset of examples drawn from the target distribution. Naturally, they perform well at generalizing within the limits of their target function, but they often fail to generalize outside of the explicitly learned feature space. It is therefore an open research topic whether and how neural network-based architectures can be deployed for systematic reasoning. Many studies have shown evidence for poor generalization, but they often work with abstract data or are limited to single-channel input. Humans, however, learn and interact through a combination of multiple sensory modalities, and rarely rely on just one. To investigate compositional generalization in a multimodal setting, we generate an extensible dataset with multimodal input sequences from simulation. We investigate the influence of the underlying training data distribution on compostional generalization in a minimal LSTM-based network trained in a supervised, time continuous setting. We find compositional generalization to fail in simple setups while improving with the number of objects, actions, and particularly with a lot of color overlaps between objects. Furthermore, multimodality strongly improves compositional generalization in settings where a pure vision model struggles to generalize.