Code quality assessment using transformers
This work addresses the need for automated evaluation of code maintainability and elegance for educators and developers, though it is incremental as it applies existing transformer methods to a new domain-specific dataset.
The paper tackled the problem of automatically assessing subjective code quality in Java programming assignments, which is challenging due to factors like excessive branching and poor naming, and found that transformer-based models like CodeBERT with task-adapted pre-training can predict quality scores more efficiently than other techniques.
Automatically evaluate the correctness of programming assignments is rather straightforward using unit and integration tests. However, programming tasks can be solved in multiple ways, many of which, although correct, are inelegant. For instance, excessive branching, poor naming or repetitiveness make the code hard to understand and maintain. These subjective qualities of code are hard to automatically assess using current techniques. In this work we investigate the use of CodeBERT to automatically assign quality score to Java code. We experiment with different models and training paradigms. We explore the accuracy of the models on a novel dataset for code quality assessment. Finally, we assess the quality of the predictions using saliency maps. We find that code quality to some extent is predictable and that transformer based models using task adapted pre-training can solve the task more efficiently than other techniques.