Measuring Improvement of F$_1$-Scores in Detection of Self-Admitted Technical Debt
This work addresses the detection of technical debt in software development, which is an incremental improvement using a novel method for a known bottleneck in NLP tasks.
The paper tackled the problem of detecting Self-Admitted Technical Debt (SATD) in software engineering by proposing a BERT-based model, which improved over previous methods in cross-project scenarios for 19 out of 20 projects, but data augmentation was insufficient for intra-project scenarios due to data limitations.
Artificial Intelligence and Machine Learning have witnessed rapid, significant improvements in Natural Language Processing (NLP) tasks. Utilizing Deep Learning, researchers have taken advantage of repository comments in Software Engineering to produce accurate methods for detecting Self-Admitted Technical Debt (SATD) from 20 open-source Java projects' code. In this work, we improve SATD detection with a novel approach that leverages the Bidirectional Encoder Representations from Transformers (BERT) architecture. For comparison, we re-evaluated previous deep learning methods and applied stratified 10-fold cross-validation to report reliable F$_1$-scores. We examine our model in both cross-project and intra-project contexts. For each context, we use re-sampling and duplication as augmentation strategies to account for data imbalance. We find that our trained BERT model improves over the best performance of all previous methods in 19 of the 20 projects in cross-project scenarios. However, the data augmentation techniques were not sufficient to overcome the lack of data present in the intra-project scenarios, and existing methods still perform better. Future research will look into ways to diversify SATD datasets in order to maximize the latent power in large BERT models.