Entropy Minimization In Emergent Languages
This work addresses the problem of understanding and improving emergent communication systems for researchers in AI and linguistics, though it is incremental as it builds on existing setups.
The study investigated the information-theoretic complexity of languages that emerge when neural agents communicate via discrete channels to solve tasks, finding that these languages minimize entropy as much as the task allows, similar to human language, and that stronger entropy minimization increases robustness to overfitting and adversarial attacks.
There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent's inputs and the messages is minimized, within the range afforded by the need for successful communication. That is, emergent languages are (nearly) as simple as the task they are developed for allow them to be. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.