Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning
This addresses the problem of costly hyperparameter tuning for machine learning practitioners by offering a more efficient, incremental approach.
The paper tackles the computational expense and opacity of hyperparameter tuning for large models by proposing an Expert Block Framework using small LLMs, achieving performance within ~0.9 percentage points of GPT-4 across six tasks with a 10-trial budget.
Hyper-parameter Tuning (HPT) is a necessary step in machine learning (ML) pipelines but becomes computationally expensive and opaque with larger models. Recently, Large Language Models (LLMs) have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks.