Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysis
For researchers using LLMs in molecular discovery, this work highlights a critical generalization failure and proposes a mitigation strategy, though the analysis is limited to specific tasks and perturbations.
The paper investigates whether molecular LLMs generalize beyond local neighborhoods induced by sequence-based representations, finding that even a single structural edit causes substantial performance drops, revealing fragile sensitivity. In-Context Tuning partially expands the local trust region, offering a direction for stabilization.
Large Language Models (LLMs) have recently shown promise in molecular discovery, yet a gap remains between their probabilistic nature over discrete sequential tokens and the rigid topological constraints of chemical space. This raises the question of whether molecular LLMs can generalize beyond the local neighborhoods induced by their sequence-based representations. To systematically investigate this question, we introduce a Molecular Perturbation framework that generates syntax-valid structural variants of training molecules under controlled Graph Edit Distance (GED) to probe the manifold regularity of molecular LLMs. Our analysis shows that even a single edit can cause substantial performance drops on common molecular tasks, revealing a narrow local trust region and fragile sensitivity to structural changes. Since similar molecules tend to exhibit similar properties, In-Context Tuning (ICT), which anchors predictions on structurally similar molecules, offers a natural way to mitigate such fragility. Our experiments also examine whether ICT confers robustness under controlled structural perturbations, and the results suggest that it can partially expand the local trust region and offer a promising direction for stabilizing molecular LLMs against structural variation.