ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery
For researchers and practitioners using HuggingFace, this work provides an automated method to identify SOTA models, but the results are incremental as they rely on existing graph structures and LLM-based verification.
The paper addresses the challenge of automatically discovering state-of-the-art models for given datasets by leveraging artifact graphs from HuggingFace. The proposed ArtifactLinker framework uses GNNs or LLMs to rank missing model-dataset links and verifies them via LLM-based coding experiments, achieving effective SOTA discovery on a new benchmark with 14,053 artifacts and 51,337 relations.
Scientific artifacts such as models and datasets are foundations for research. With the rapid growth of platforms like HuggingFace, researchers now have access to a large number of artifacts. Yet, a key challenge remains: how can we automatically discover the state-of-the-art (SOTA) model for a given dataset by fully leveraging existing artifacts? We formalize this task as automatic SOTA discovery by modeling HuggingFace as an artifact graph, where nodes are models/datasets and edges represent evaluations. We propose ArtifactLinker, a two-stage framework: (1) ranking promising unobserved model--dataset links using Graph Neural Networks (GNNs) or graph-augmented Large Language Models (LLMs), and (2) verifying top-ranked links via coding experiments with LLM-based agents. We further introduce a benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations to evaluate the performance of both stages. Results show that (1) graph structures between existing artifacts are effective for missing link prediction; (2) end-to-end ranking and verification with ArtifactLinker help discover potential SOTA results and research insights.