Prototype Guided Post-pretraining for Single-Cell Representation Learning
For computational biologists, this addresses the long-tailed cell-type distribution and covariate shift limitations of single-cell pretrained models, though the improvement is incremental.
CellRefine, a post-pretraining method for single-cell representation learning, uses marker-gene structural priors to refine latent embeddings, achieving up to 15% improvement across multiple tasks.
Single-cell representation learning (SCRL) from gene expression data offers a way to uncover the complex regulatory logic underlying cellular function. Inspired by large language models in natural language modeling, several single-cell pretrained models have recently been proposed that treat genes as tokens and cells as sentences. However, these models are fundamentally limited by the long-tailed nature of cell-type distributions and struggle to generalize under covariate shifts in gene expression data. While fine-tuning is often used to mitigate these issues, we observe that performance remains bounded. To address this challenge, we introduce CellRefine, a post-pretraining method that operates between the pretraining and fine-tuning stages of a single-cell foundation model. CellRefine uses a multi-faceted objective that incorporates marker-gene sets as structural priors to guide post-pretraining and refine the latent embedding manifold of cells. Across multiple computational biology tasks, empirical results show that CellRefine consistently improves downstream performance, yielding gains up to 15%.