Chris Brown

Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou et al.

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixEval provides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced at https://github.com/mahimanzum/FixEval.

Arpit Thool, Chris Brown

Modern development methodologies, such as Kanban and continuous integration and continuous deployment (CI/CD), are critical for web application development -- as software products must adapt to changing requirements and deploy products to users quickly. As web application attacks and exploited vulnerabilities are rising, it is increasingly crucial to integrate security into modern development practices. Yet, the iterative and incremental nature of these processes can clash with the sequential nature of security engineering. Thus, it is challenging to adopt security practices and activities in modern development practices. Dynamic Application Security Testing (DAST) is a security practice within software development frameworks that bolsters system security. This study delves into the intersection of Agile development and DAST, exploring how a software organization attempted to integrate DAST into their Kanban workflows and CI/CD pipelines to identify and mitigate security vulnerabilities within the development process. Through an action research case study incorporating interviews among team members, this research elucidates the challenges, mitigation techniques, and best practices associated with incorporating DAST into Agile methodologies from developers' perspectives. We provide insights into integrating security practices with modern development, ensuring both speed and security in software delivery.

Peipei Wang, Chris Brown, Jamie A. Jennings et al.

Regular expressions cause string-related bugs and open security vulnerabilities for DOS attacks. However, beyond ReDoS (Regular expression Denial of Service), little is known about the extent to which regular expression issues affect software development and how these issues are addressed in practice. We conduct an empirical study of 356 merged regex-related pull request bugs from Apache, Mozilla, Facebook, and Google GitHub repositories. We identify and classify the nature of the regular expression problems, the fixes, and the related changes in the test code. The most important findings in this paper are as follows: 1) incorrect regular expression behavior is the dominant root cause of regular expression bugs (165/356, 46.3%). The remaining root causes are incorrect API usage (9.3%) and other code issues that require regular expression changes in the fix (29.5%), 2) fixing regular expression bugs is nontrivial as it takes more time and more lines of code to fix them compared to the general pull requests, 3) most (51%) of the regex-related pull requests do not contain test code changes. Certain regex bug types (e.g., compile error, performance issues, regex representation) are less likely to include test code changes than others, and 4) the dominant type of test code changes in regex-related pull requests is test case addition (75%). The results of this study contribute to a broader understanding of the practical problems faced by developers when using, fixing, and testing regular expressions.

Chris Brown, Chris Parnin

Student experiences in large undergraduate Computer Science courses are increasingly impacted by automated systems. Bots, or agents of software automation, are useful for efficiently grading and generating feedback. Current efforts at automation in CS education focus on supporting instructional tasks, but do not address student struggles due to poor behaviors, such as procrastination. In this paper, we explore using bots to improve the software engineering behaviors of students using developer recommendation choice architectures, a framework incorporating behavioral science concepts in recommendations to improve the actions of programmers. We implemented this framework in class-bot, a novel system designed to nudge students to make better choices while working on programming assignments. This work presents a preliminary evaluation integrating this tool in an introductory programming course. Our results show that class-bot is beneficial for improving student development behaviors increasing code quality and productivity.