ViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks
This benchmark provides a standardized evaluation framework for NFMs in viral genomics, addressing both biological understanding and biosecurity risks, which is critical for the biomedical community.
ViroBench is the first comprehensive benchmark for nucleotide foundation models (NFMs) on viral genomics, evaluating 66 models across 18 scenarios. Key findings include performance degradation under phylogenetic/temporal shifts, decoupling of likelihood and functional validity in generation tasks, and that taxonomic diversity in pretraining data outweighs parameter scale, with a lightweight baseline achieving 67.5% performance gain.
Nucleotide sequences constitute the fundamental genetic basis of biological systems, rendering viral genomic analysis critical for biomedical advancement. Despite progress in biological foundation models, specifically nucleotide foundation models (NFMs), the field lacks a unified standard for viral genomics to facilitate community development and enforce biosecurity constraints. To address this, we introduce ViroBench, the first comprehensive and large-scale benchmark specifically designed for NFMs in viral settings. ViroBench evaluates models across two critical dimensions: biological understanding and latent biosecurity risk, covering 18 diverse scenarios within 4 task types. Extensive evaluation of 66 NFMs across diverse architectures yields three critical conclusions. Firstly, NFMs exhibit a performance degradation in biological understanding under phylogenetic and temporal shifts, indicating weak extrapolation capabilities. Secondly, generation tasks reveal a decoupling between statistical likelihood and biological functional validity, posing latent biosecurity risks. Thirdly, controlled ablation studies reveal that taxonomic diversity in pretraining data outweighs parameter scale. Specifically, a lightweight baseline trained on diverse data achieves a 67.5% performance gain over its original model. Overall, ViroBench provides interpretable, diagnostic evaluations and a reproducible measurement framework for future research on viral nucleotide foundation models. The datasets and code are publicly available at https://github.com/QIANJINYDX/ViroBench.