XferBench: a Data-Driven Benchmark for Emergent Language
This provides a tool for researchers in AI and linguistics to assess emergent languages, though it is incremental as it builds on existing methods for language evaluation.
The authors tackled the problem of evaluating emergent languages by introducing XferBench, a data-driven benchmark that measures language quality based on downstream NLP task performance, achieving validation with human, synthetic, and emergent baselines.
In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines.