How Software Engineering Students Use LLMs to Write Research Papers: An Experience Report
For educators in software engineering, this provides insights into how students actually use LLMs in empirical assignments, but the findings are incremental and context-specific.
This experience report analyzes how 146 software engineering students used LLMs in writing research papers for an empirical methods assignment, finding they used LLMs for brainstorming, methodological clarification, organization, and writing refinement, while also reporting concerns about inaccuracies and content verification.
Large language models are increasingly becoming part of software engineering education, including activities involving empirical software engineering and evidence synthesis. This paper reports an educational experience involving the integration of reflective LLM use into an empirical methods assignment in a third-year software architecture course. Students were asked to develop a short research paper using either a rapid review or a gray literature review methodology and to disclose how LLMs were used throughout the assignment. We analyzed 146 student disclosure statements using a cross-analysis process combining LLM-assisted categorization with manual verification and refinement by the researchers. The reflections describe how students incorporated LLMs during activities such as brainstorming, methodological clarification, organization of findings, and writing refinement, while also reporting concerns regarding inaccuracies and verification of generated content. This experience report discusses lessons learned and educational implications for integrating AI-assisted technologies into empirical software engineering education.