BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks
This provides a cost-effective solution for researchers and practitioners automating ML tasks with LLMs, though it's an incremental improvement over existing single-agent systems.
The paper tackles the problem of high costs when using large LLMs like GPT-4 for automating complex machine learning tasks, proposing a multi-agent system that combines no-cost models with occasional GPT-4 calls to achieve a 94.2% cost reduction (from $0.931 to $0.054 per run) while improving average success rate from 22.72% to 32.95% on the MLAgentBench benchmark.
Large Language Models (LLMs) excel in diverse applications including generation of code snippets, but often struggle with generating code for complex Machine Learning (ML) tasks. Although existing LLM single-agent based systems give varying performance depending on the task complexity, they purely rely on larger and expensive models such as GPT-4. Our investigation reveals that no-cost and low-cost models such as Gemini-Pro, Mixtral and CodeLlama perform far worse than GPT-4 in a single-agent setting. With the motivation of developing a cost-efficient LLM based solution for solving ML tasks, we propose an LLM Multi-Agent based system which leverages combination of experts using profiling, efficient retrieval of past observations, LLM cascades, and ask-the-expert calls. Through empirical analysis on ML engineering tasks in the MLAgentBench benchmark, we demonstrate the effectiveness of our system, using no-cost models, namely Gemini as the base LLM, paired with GPT-4 in cascade and expert to serve occasional ask-the-expert calls for planning. With 94.2\% reduction in the cost (from \$0.931 per run cost averaged over all tasks for GPT-4 single agent system to \$0.054), our system is able to yield better average success rate of 32.95\% as compared to GPT-4 single-agent system yielding 22.72\% success rate averaged over all the tasks of MLAgentBench.