Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?
This addresses the problem of resource-intensive hyperparameter tuning for researchers and practitioners in computer vision, offering a more efficient alternative, though it appears incremental as it adapts existing LLM techniques to a specific application.
This work tackles hyperparameter optimization for neural networks in computer vision by fine-tuning Code Llama with LoRA to generate recommendations, achieving competitive or superior RMSE compared to traditional methods like Optuna while reducing computational overhead.
Optimal hyperparameter selection is critical for maximizing the performance of neural networks in computer vision, particularly as architectures become more complex. This work explores the use of large language models (LLMs) for hyperparameter optimization by fine-tuning a parameter-efficient version of Code Llama using LoRA. The resulting model produces accurate and computationally efficient hyperparameter recommendations across a wide range of vision architectures. Unlike traditional methods such as Optuna, which rely on resource-intensive trial-and-error procedures, our approach achieves competitive or superior Root Mean Square Error (RMSE) while substantially reducing computational overhead. Importantly, the models evaluated span image-centric tasks such as classification, detection, and segmentation, fundamental components in many image manipulation pipelines including enhancement, restoration, and style transfer. Our results demonstrate that LLM-based optimization not only rivals established Bayesian methods like Tree-structured Parzen Estimators (TPE), but also accelerates tuning for real-world applications requiring perceptual quality and low-latency processing. All generated configurations are publicly available in the LEMUR Neural Network Dataset (https://github.com/ABrain-One/nn-dataset), which serves as an open source benchmark for hyperparameter optimization research and provides a practical resource to improve training efficiency in image manipulation systems.