LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
This addresses the challenge of model and hyperparameter selection for machine learning practitioners, offering a lightweight, general-purpose assistant, though it appears incremental as it builds on existing LLM capabilities.
The paper tackled the problem of model and hyperparameter selection in machine learning by investigating whether large language models (LLMs) can act as in-context meta-learners for this task, showing that LLMs can recommend competitive models and hyperparameters without search across synthetic and real-world benchmarks.
Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.