Rethinking Molecular Text Representations for LLMs: An Empirical Study
This work provides a comprehensive empirical study that reveals the critical role of molecular representation choice for LLMs, guiding task-aware representation selection in cheminformatics.
The paper benchmarks 16 LLMs across nine molecular representations and eight chemical tasks, finding that performance is strongly representation-dependent with no single representation winning across all tasks; CML and MolJSON excel at structural tasks, IUPAC at semantic tasks, and SMILES variants are rarely optimal despite their prevalence in pretraining.
Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark evaluating LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI, and then canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a-judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes and attention shows that representations are encoded differently inside the model; for example, structured representations require higher attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.