Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability
This work addresses the challenge of designing more effective emergent communication systems for AI agents, though it is incremental as it builds on existing language game frameworks.
The paper tackles the problem of understanding what determines the expressivity of emergent languages in deep learning models, showing that it involves a trade-off between contextual complexity and unpredictability, and discovers message type collapse where unique messages are fewer than inputs.
Researchers are using deep learning models to explore the emergence of language in various language games, where agents interact and develop an emergent language to solve tasks. We focus on the factors that determine the expressivity of emergent languages, which reflects the amount of information about input spaces those languages are capable of encoding. We measure the expressivity of emergent languages based on the generalisation performance across different games, and demonstrate that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context those languages emerged from. Another contribution of this work is the discovery of message type collapse, i.e. the number of unique messages is lower than that of inputs. We also show that using the contrastive loss proposed by Chen et al. (2020) can alleviate this problem.