Caio Barbosa

Joao Correia, Daniel Coutinho, Marco Castelluccio et al.

The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.

Filipe Falcão, Caio Barbosa, Baldoino Fonseca et al.

[This paper has been withdrawn by the author due to updated research available on arXiv (arXiv:1811.01918)] As the modern open-source paradigm makes it easier to contribute to software projects, the number of developers involved in these projects keep increasing. This growth in the amount of developers makes it more difficult to deal with harmful contributions. Recent researches have found that technical and social factors can predict the success of contributions to open-source projects on GitHub. However, these researches do not study the relation between these factors with the introduction of bugs. Our study aims at investigating the influence of technical (such as, developers' experience) and social (such as, number of followers) factors on the introduction of bugs, using information from 14 projects hosted on GitHub. Understanding the influence of these factors may be useful to developers, code reviewers and researchers. For instance, code reviewers may want to double check commits from developers that present bug-related factors. We found that technical factors have a consistent influence in the introduction of bugs. On the other hand, social factors present signs of influence in bug introduction that would require more data to be properly evaluated. Moreover, we found that perils present in the mining of GitHub may impact the factors results.

Filipe Falcão, Caio Barbosa, Baldoino Fonseca et al.

As collaborative coding environments make it easier to contribute to software projects, the number of developers involved in these projects keeps increasing. This increase makes it more difficult for code reviewers to deal with buggy contributions. Collaborative environments like GitHub provide a rich source of data on developers' contributions. Such data can be used to extract information about developers regarding technical (e.g., their experience) and social (e.g., their interactions) factors. Recent studies analyzed the influence of these factors on different activities of software development. However, there is still room for improvement on the relation between these factors and the introduction of bugs. We present a broader study, including 8 projects from different domains and 6,537 bug reports, on relating five technical, three social factors, and the introduction of bugs. The results indicate that technical and social factors can discriminate between buggy and clean commits. But, the technical factors are more determining than social ones. Particularly, the developers' habits of not following technical contribution norms and the developer's commit bugginess are associated with an increase on commit bugginess. On the other hand, project's establishment, ownership level of developers' commit, and social influence are related to a lower chance of introducing bugs.