A Neural Question Answering System for Basic Questions about Subroutines
This work addresses the problem of providing quick answers to basic programming questions for software engineers, laying groundwork for future dialogue systems in software engineering.
This paper introduces a neural question-answering system designed to answer basic questions about subroutines. The system was trained on a dataset of 10.9 million question/context/answer tuples and evaluated with professional programmers, demonstrating its capabilities and limitations.
A question answering (QA) system is a type of conversational AI that generates natural language answers to questions posed by human users. QA systems often form the backbone of interactive dialogue systems, and have been studied extensively for a wide variety of tasks ranging from restaurant recommendations to medical diagnostics. Dramatic progress has been made in recent years, especially from the use of encoder-decoder neural architectures trained with big data input. In this paper, we take initial steps to bringing state-of-the-art neural QA technologies to Software Engineering applications by designing a context-based QA system for basic questions about subroutines. We curate a training dataset of 10.9 million question/context/answer tuples based on rules we extract from recent empirical studies. Then, we train a custom neural QA model with this dataset and evaluate the model in a study with professional programmers. We demonstrate the strengths and weaknesses of the system, and lay the groundwork for its use in eventual dialogue systems for software engineering.