Discovering Hidden Gems in Model Repositories
This addresses the inefficiency in model selection for AI practitioners by identifying overlooked high-performing models, though it is incremental as it builds on existing search algorithms.
The study tackled the problem of underutilized superior models in public repositories by evaluating over 2,000 models, finding 'hidden gems' that significantly outperform popular ones, such as improving math performance from 83.2% to 96.0% in the Llama-3.1-8B family. They developed a Multi-Armed Bandit-based method that accelerates discovery by over 50x, using as few as 50 queries per candidate.
Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.