dnaHNet: A Scalable and Hierarchical Foundation Model for Genomic Sequence Learning
This work addresses a fundamental bottleneck in genomic sequence modeling for researchers, offering a scalable and interpretable framework that balances efficiency with biological accuracy.
The paper tackled the tradeoff between biological coherence and computational cost in genomic foundation models by introducing dnaHNet, a tokenizer-free autoregressive model that uses differentiable dynamic chunking to adaptively compress raw nucleotides, achieving >3x inference speedup over Transformers and superior zero-shot performance in tasks like protein variant fitness prediction.
Genomic foundation models have the potential to decode DNA syntax, yet face a fundamental tradeoff in their input representation. Standard fixed-vocabulary tokenizers fragment biologically meaningful motifs such as codons and regulatory elements, while nucleotide-level models preserve biological coherence but incur prohibitive computational costs for long contexts. We introduce dnaHNet, a state-of-the-art tokenizer-free autoregressive model that segments and models genomic sequences end-to-end. Using a differentiable dynamic chunking mechanism, dnaHNet compresses raw nucleotides into latent tokens adaptively, balancing compression with predictive accuracy. Pretrained on prokaryotic genomes, dnaHNet outperforms leading architectures including StripedHyena2 in scaling and efficiency. This recursive chunking yields quadratic FLOP reductions, enabling $>3 \times$ inference speedup over Transformers. On zero-shot tasks, dnaHNet achieves superior performance in predicting protein variant fitness and gene essentiality, while automatically discovering hierarchical biological structures without supervision. These results establish dnaHNet as a scalable, interpretable framework for next-generation genomic modeling.