On Assessing the Relevance of Code Reviews Authored by Generative Models
This addresses the need for better evaluation methods in code review automation, though it is incremental in improving assessment techniques.
The paper tackles the problem of evaluating code reviews generated by large language models like ChatGPT, proposing a multi-subjective ranking method that uses human judges to rank comments. Results show ChatGPT's comments were ranked significantly better than human ones, even surpassing accepted answers on CodeReview StackExchange.
The use of large language models like ChatGPT in code review offers promising efficiency gains but also raises concerns about correctness and safety. Existing evaluation methods for code review generation either rely on automatic comparisons to a single ground truth, which fails to capture the variability of human perspectives, or on subjective assessments of "usefulness", a highly ambiguous concept. We propose a novel evaluation approach based on what we call multi-subjective ranking. Using a dataset of 280 self-contained code review requests and corresponding comments from CodeReview StackExchange, multiple human judges ranked the quality of ChatGPT-generated comments alongside the top human responses from the platform. Results show that ChatGPT's comments were ranked significantly better than human ones, even surpassing StackExchange's accepted answers. Going further, our proposed method motivates and enables more meaningful assessments of generative AI's performance in code review, while also raising awareness of potential risks of unchecked integration into review processes.