NLU for Game-based Learning in Real: Initial Evaluations
This work addresses the challenge of deploying conversational agents in educational games for children, but it is incremental as it applies an existing transformer-based method to new, small datasets.
The study tackled the problem of improving Natural Language Understanding (NLU) for goal-oriented spoken dialogue systems in real-world game-based learning, showing that the Dual Intent and Entity Transformer (DIET) architecture is robust for intent recognition on domain-specific datasets, with observed performance drops in-the-wild.
Intelligent systems designed for play-based interactions should be contextually aware of the users and their surroundings. Spoken Dialogue Systems (SDS) are critical for these interactive agents to carry out effective goal-oriented communication with users in real-time. For the real-world (i.e., in-the-wild) deployment of such conversational agents, improving the Natural Language Understanding (NLU) module of the goal-oriented SDS pipeline is crucial, especially with limited task-specific datasets. This study explores the potential benefits of a recently proposed transformer-based multi-task NLU architecture, mainly to perform Intent Recognition on small-size domain-specific educational game datasets. The evaluation datasets were collected from children practicing basic math concepts via play-based interactions in game-based learning settings. We investigate the NLU performances on the initial proof-of-concept game datasets versus the real-world deployment datasets and observe anticipated performance drops in-the-wild. We have shown that compared to the more straightforward baseline approaches, Dual Intent and Entity Transformer (DIET) architecture is robust enough to handle real-world data to a large extent for the Intent Recognition task on these domain-specific in-the-wild game datasets.