LLM-Crowdsourced: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
This addresses the problem of comprehensive and objective evaluation of LLMs for researchers and developers, though it is incremental as it builds on existing evaluation methods by introducing a new paradigm.
The authors tackled the challenge of evaluating large language models (LLMs) by proposing LLM-Crowdsourced, a benchmark-free paradigm that uses LLMs to generate questions, answer independently, and evaluate mutually, which effectively distinguished performance across eight models in mathematics and programming, revealing findings like Gemini's superior question-design capabilities and high consistency in evaluations.
Although large language models (LLMs) demonstrate remarkable capabilities across various tasks, evaluating their capabilities remains a challenging task. Existing evaluation methods suffer from issues such as data contamination, black-box operation, and subjective preference. These issues make it difficult to evaluate the LLMs' true capabilities comprehensively. To tackle these challenges, we propose a novel benchmark-free evaluation paradigm, LLM-Crowdsourced. It utilizes LLMs to generate questions, answer independently, and evaluate mutually. This method integrates four key evaluation criteria: dynamic, transparent, objective, and professional, which existing evaluation methods cannot satisfy simultaneously. Experiments on eight mainstream LLMs across mathematics and programming verify the advantages of our method in distinguishing LLM performance. Furthermore, our study reveals several novel findings that are difficult for traditional methods to detect, including but not limited to: (1) Gemini demonstrates the highest original and professional question-design capabilities among others; (2) Some LLMs exhibit ''memorization-based answering'' by misrecognizing questions as familiar ones with a similar structure; (3) LLM evaluation results demonstrate high consistency (robustness).