Cross-Granularity Representations for Biological Sequences: Insights from ESM and BiGCARP
This addresses the challenge of improving performance and interpretability in biological foundation models, though it appears incremental as it builds on existing models like ESM and BiGCARP.
The paper tackled the problem of integrating cross-granularity knowledge from biological sequence models, showing that combining representations from amino acid-level and domain-level models yields measurable performance gains in intermediate-level prediction tasks.
Recent advances in general-purpose foundation models have stimulated the development of large biological sequence models. While natural language shows symbolic granularity (characters, words, sentences), biological sequences exhibit hierarchical granularity whose levels (nucleotides, amino acids, protein domains, genes) further encode biologically functional information. In this paper, we investigate the integration of cross-granularity knowledge from models through a case study of BiGCARP, a Pfam domain-level model for biosynthetic gene clusters, and ESM, an amino acid-level protein language model. Using representation analysis tools and a set of probe tasks, we first explain why a straightforward cross-model embedding initialization fails to improve downstream performance in BiGCARP, and show that deeper-layer embeddings capture a more contextual and faithful representation of the model's learned knowledge. Furthermore, we demonstrate that representations at different granularities encode complementary biological knowledge, and that combining them yields measurable performance gains in intermediate-level prediction tasks. Our findings highlight cross-granularity integration as a promising strategy for improving both the performance and interpretability of biological foundation models.