GRAFITE: Generative Regression Analysis Framework for Issue Tracking and Evaluation
This addresses the risk of inflated LLM performance due to benchmark contamination, providing a practical tool for ongoing model evaluation.
The authors tackled the problem of LLM benchmark contamination by developing GRAFITE, a continuous evaluation platform that builds a repository of model issues from user feedback and assesses LLMs against these issues using QA tests with LLM-as-a-judge, enabling side-by-side model comparisons and regression detection.
Large language models (LLMs) are largely motivated by their performance on popular topics and benchmarks at the time of their release. However, over time, contamination occurs due to significant exposure of benchmark data during training. This poses a risk of model performance inflation if testing is not carefully executed. To address this challenge, we present GRAFITE, a continuous LLM evaluation platform through a comprehensive system for maintaining and evaluating model issues. Our approach enables building a repository of model problems based on user feedback over time and offers a pipeline for assessing LLMs against these issues through quality assurance (QA) tests using LLM-as-a-judge. The platform enables side-by-side comparison of multiple models, facilitating regression detection across different releases. The platform is available at https://github.com/IBM/grafite. The demo video is available at www.youtube.com/watch?v=XFZyoleN56k.