Mathematically Modeling the Lexicon Entropy of Emergent Language
This provides a mathematical model for analyzing emergent language in AI systems, though it appears incremental as it builds on existing work in this niche area.
The researchers tackled the problem of predicting lexicon entropy in deep learning-based emergent language systems by formulating a stochastic process called FiLex, which correctly predicted correlations between hyperparameters and entropy in all 20 tested environment-hyperparameter combinations.
We formulate a stochastic process, FiLex, as a mathematical model of lexicon entropy in deep learning-based emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and decisively tested. We empirically verify across four different environments that FiLex predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature) and the emergent language's entropy in 20 out of 20 environment-hyperparameter combinations. Furthermore, our experiments reveal that different environments show diverse relationships between their hyperparameters and entropy which demonstrates the need for a model which can make well-defined predictions at a precise level of granularity.