A single-cell gene expression language model
This work provides a framework for modeling gene regulation at the single-cell level, which could aid in understanding genotype-phenotype connections, but it is incremental as it applies existing language modeling techniques to a new biological domain.
The authors tackled the problem of learning gene regulatory rules from single-cell RNA expression data by proposing Exceiver, a language model that learns context dependencies between genes, and found that pretraining supports transfer learning to downstream tasks.
Gene regulation is a dynamic process that connects genotype and phenotype. Given the difficulty of physically mapping mammalian gene circuitry, we require new computational methods to learn regulatory rules. Natural language is a valuable analogy to the communication of regulatory control. Machine learning systems model natural language by explicitly learning context dependencies between words. We propose a similar system applied to single-cell RNA expression profiles to learn context dependencies between genes. Our model, Exceiver, is trained across a diversity of cell types using a self-supervised task formulated for discrete count data, accounting for feature sparsity. We found agreement between the similarity profiles of latent sample representations and learned gene embeddings with respect to biological annotations. We evaluated Exceiver on a new dataset and a downstream prediction task and found that pretraining supports transfer learning. Our work provides a framework to model gene regulation on a single-cell level and transfer knowledge to downstream tasks.