Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systems
This work addresses the problem of perceptual heterogeneity in multi-agent systems for researchers studying emergent communication, positioning it as a framework for studying representation transfer across modalities.
The researchers studied how agents with different perceptual modalities develop communication in a multi-step binary game, finding that multimodal systems converge to class-consistent messages despite perceptual misalignment, with unimodal systems communicating more efficiently using fewer bits and achieving lower classification entropy.
Emergent communication offers insight into how agents develop shared structured representations, yet most research assumes homogeneous modalities or aligned representational spaces, overlooking the perceptual heterogeneity of real-world settings. We study a heterogeneous multi-step binary communication game where agents differ in modality and lack perceptual grounding. Despite perceptual misalignment, multimodal systems converge to class-consistent messages grounded in perceptual input. Unimodal systems communicate more efficiently, using fewer bits and achieving lower classification entropy, while multimodal agents require greater information exchange and exhibit higher uncertainty. Bit perturbation experiments provide strong evidence that meaning is encoded in a distributional rather than compositional manner, as each bit's contribution depends on its surrounding pattern. Finally, interoperability analyses show that systems trained in different perceptual worlds fail to directly communicate, but limited fine-tuning enables successful cross-system communication. This work positions emergent communication as a framework for studying how agents adapt and transfer representations across heterogeneous modalities, opening new directions for both theory and experimentation.