Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding
This work addresses the need for standardized evaluation in on-the-job learning for dialogue systems, which is incremental as it builds on existing systems but introduces a novel methodology.
The paper tackles the lack of a general evaluation methodology for on-the-job learning dialogue systems by proposing a first such methodology and applying it to a case study where a task-oriented dialogue system improves its natural language understanding component through user interactions.
On-the-job learning consists in continuously learning while being used in production, in an open environment, meaning that the system has to deal on its own with situations and elements never seen before. The kind of systems that seem to be especially adapted to on-the-job learning are dialogue systems, since they can take advantage of their interactions with users to collect feedback to adapt and improve their components over time. Some dialogue systems performing on-the-job learning have been built and evaluated but no general methodology has yet been defined. Thus in this paper, we propose a first general methodology for evaluating on-the-job learning dialogue systems. We also describe a task-oriented dialogue system which improves on-the-job its natural language component through its user interactions. We finally evaluate our system with the described methodology.