Leveraging Generative AI: Improving Software Metadata Classification with Generated Code-Comment Pairs
This work addresses a domain-specific challenge for software developers and researchers in NLP and software engineering, but it is incremental as it applies existing methods to new data.
The paper tackled the problem of classifying code comments as 'Useful' or 'Not Useful' by using contextualized embeddings like BERT and augmenting a dataset with generated code-comment pairs, achieving evaluation with precision, recall, and F1-score metrics across various machine learning algorithms.
In software development, code comments play a crucial role in enhancing code comprehension and collaboration. This research paper addresses the challenge of objectively classifying code comments as "Useful" or "Not Useful." We propose a novel solution that harnesses contextualized embeddings, particularly BERT, to automate this classification process. We address this task by incorporating generated code and comment pairs. The initial dataset comprised 9048 pairs of code and comments written in C, labeled as either Useful or Not Useful. To augment this dataset, we sourced an additional 739 lines of code-comment pairs and generated labels using a Large Language Model Architecture, specifically BERT. The primary objective was to build classification models that can effectively differentiate between useful and not useful code comments. Various machine learning algorithms were employed, including Logistic Regression, Decision Tree, K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Gradient Boosting, Random Forest, and a Neural Network. Each algorithm was evaluated using precision, recall, and F1-score metrics, both with the original seed dataset and the augmented dataset. This study showcases the potential of generative AI for enhancing binary code comment quality classification models, providing valuable insights for software developers and researchers in the field of natural language processing and software engineering.