CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
This addresses the challenge of complex code deployment for computer science researchers, though it appears incremental as it applies existing LLM capabilities to a specific domain task.
The authors tackled the problem of automating deployment of computer science research code repositories by introducing CSR-Bench, a benchmark for evaluating LLM agents, and CSR-Agents, a framework using multiple LLM agents to generate and improve bash commands for repository deployment. Preliminary results show LLM agents can significantly enhance deployment workflows and boost developer productivity.
The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.