An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspective
This work addresses the problem of unclear definitions and usage challenges for DevBots in software development, providing a framework for future research, but it is incremental as it builds on existing studies without introducing new methods.
The study tackled the lack of consensus on software engineering bots (DevBots) by conducting a mixed-method empirical study with 21 interviews and 111 surveys of developers, identifying three user personas with different definitions and challenges.
Software engineering bots - automated tools that handle tedious tasks - are increasingly used by industrial and open source projects to improve developer productivity. Current research in this area is held back by a lack of consensus of what software engineering bots (DevBots) actually are, what characteristics distinguish them from other tools, and what benefits and challenges are associated with DevBot usage. In this paper we report on a mixed-method empirical study of DevBot usage in industrial practice. We report on findings from interviewing 21 and surveying a total of 111 developers. We identify three different personas among DevBot users (focusing on autonomy, chat interfaces, and "smartness"), each with different definitions of what a DevBot is, why developers use them, and what they struggle with. We conclude that future DevBot research should situate their work within our framework, to clearly identify what type of bot the work targets, and what advantages practitioners can expect. Further, we find that there currently is a lack of general purpose "smart" bots that go beyond simple automation tools or chat interfaces. This is problematic, as we have seen that such bots, if available, can have a transformative effect on the projects that use them.