RoCar: A Relationship Network-based Evaluation Method for Large Language Models
This addresses the challenge of fairly evaluating LLMs for researchers and developers, though it appears incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating large language models (LLMs) by proposing the RoCar method, which uses random task graphs to generate natural language tasks for assessing reasoning and memory abilities, ensuring fairness by preventing direct learning of evaluation tasks.
Large language models (LLMs) have received increasing attention. However, due to the complexity of its capabilities, how to rationally evaluate the capabilities of LLMs is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph and generates natural language evaluation tasks based on the task graph to evaluate the reasoning and memory abilities of LLMs respectively. Due to the very large randomness of the task construction process, it is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method.