Propensity Score Alignment of Unpaired Multimodal Data
This addresses the challenge of learning from unpaired samples in fields like biology, where paired data is hard to collect, offering a novel solution for multimodal representation learning.
The paper tackles the problem of aligning unpaired multimodal data by drawing an analogy to causal inference to estimate a common space, and finds that optimal transport matching significantly improves over state-of-the-art methods in synthetic and real-world datasets.
Multimodal representation learning techniques typically rely on paired samples to learn common representations, but paired samples are challenging to collect in fields such as biology where measurement devices often destroy the samples. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning. We draw an analogy between potential outcomes in causal inference and potential views in multimodal observations, which allows us to use Rubin's framework to estimate a common space in which to match samples. Our approach assumes we collect samples that are experimentally perturbed by treatments, and uses this to estimate a propensity score from each modality, which encapsulates all shared information between a latent state and treatment and can be used to define a distance between samples. We experiment with two alignment techniques that leverage this distance -- shared nearest neighbours (SNN) and optimal transport (OT) matching -- and find that OT matching results in significant improvements over state-of-the-art alignment approaches in both a synthetic multi-modal setting and in real-world data from NeurIPS Multimodal Single-Cell Integration Challenge.