Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matching
For researchers with limited compute, this provides an efficient way to obtain high-quality mRNA representations without running large models.
The authors distilled a large genomic foundation model (billions of parameters) into a 200× smaller model specialized for mRNA sequences using embedding-level distillation, achieving state-of-the-art performance among similarly sized models on mRNA-bench and competing with larger architectures.
Large Genomic Foundation Models have recently achieved remarkable results and in-vivo translation capabilities. However these models quickly grow to over a few Billion of parameters and are expensive to run when compute is limited. To overcome this challenge, we present a distillation framework for transferring mRNA representations from a state of the art genomic foundation model into a much smaller model specialized for mRNA sequences, reducing the size by 200-fold. Embedding-level distillation worked better than logit based methods, which we found unstable. Benchmarking on mRNA-bench demonstrates that the distilled model achieves state-of-the-art performance among models of comparable size and competes with larger architectures for mRNA-related tasks. Our results highlight embedding-based distillation of mRNA sequences as an effective training strategy for biological foundation models. This enables similar efficient and scalable sequence modelling in genomics, particularly when large models are computationally challenging or infeasible.