TEDDY: A Family Of Foundation Models For Understanding Single Cell Biology
This work addresses the challenge of improving foundation models for single-cell biology analysis, which is important for drug discovery and medicine, though it appears incremental in its approach.
The researchers tackled the problem of limited performance of existing single-cell foundation models in downstream applications by scaling pre-training to 116 million cells and incorporating biological annotations as supervision, resulting in TEDDY models that showed substantial improvement in identifying disease states of unseen donors and more muted improvements in distinguishing healthy from diseased cells.
Understanding the biological mechanism of disease is critical for medicine, and in particular drug discovery. AI-powered analysis of genome-scale biological data hold great potential in this regard. The increasing availability of single-cell RNA sequencing data has enabled the development of large foundation models for disease biology. However, existing foundation models either do not improve or only modestly improve over task-specific models in downstream applications. Here, we explored two avenues for improving the state-of-the-art. First, we scaled the pre-training dataset to 116 million cells, which is larger than those used by previous models. Second, we leveraged the availability of large-scale biological annotations as a form of supervision during pre-training. We trained the TEDDY family of models comprising six transformer-based state-of-the-art single-cell foundation models with 70 million, 160 million, and 400 million parameters. We vetted our models on two downstream evaluation tasks -- identifying the underlying disease state of held-out donors not seen during training and distinguishing healthy cells from diseased ones for disease conditions and donors not seen during training. Scaling experiments showed that performance improved predictably with both data volume and parameter count. Our models showed substantial improvement over existing work on the first task and more muted improvements on the second.