Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language
This addresses the challenge of language acquisition without prior knowledge for AI systems, though it is incremental as it focuses on a specific domain.
The paper tackles the problem of learning language interpretation and generation using only perceptual context as supervision, resulting in a system that can sportscast simulated robot soccer games in English and Korean with human-evaluated reasonable quality, sometimes on par with human performance in the limited domain.
We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.