Freeze, Diffuse, Decode: Geometry-Aware Adaptation of Pretrained Transformer Embeddings for Antimicrobial Peptide Design
This work addresses a bottleneck in transfer learning for tasks like antimicrobial peptide design, offering a novel method to preserve embedding geometry, but it is incremental as it builds on existing diffusion-based and transformer techniques.
The paper tackled the problem of adapting pretrained transformer embeddings to downstream tasks without distorting their geometric structure, especially when supervised data is scarce, and introduced the Freeze, Diffuse, Decode (FDD) framework, which achieved predictive and interpretable representations for antimicrobial peptide design.
Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: fine-tuning and probing, either distort the pretrained geometric structure of the embeddings or lack sufficient expressivity to capture task-relevant signals. These issues become even more pronounced when supervised data are scarce. Here, we introduce Freeze, Diffuse, Decode (FDD), a novel diffusion-based framework that adapts pre-trained embeddings to downstream tasks while preserving their underlying geometric structure. FDD propagates supervised signal along the intrinsic manifold of frozen embeddings, enabling a geometry-aware adaptation of the embedding space. Applied to antimicrobial peptide design, FDD yields low-dimensional, predictive, and interpretable representations that support property prediction, retrieval, and latent-space interpolation.