Molecular Geometry-aware Transformer for accurate 3D Atomic System modeling
This work addresses the need for faster and more accurate molecular simulations in physics, chemistry, and biology, offering an incremental improvement over previous Transformer models by explicitly modeling geometric factors.
The paper tackles the problem of predicting molecular energy and properties by proposing Moleformer, a Transformer architecture that incorporates geometry-aware spatial encoding for atoms, bonds, and nonbonding pairs, achieving state-of-the-art results on OC20 for energy prediction and competitive performance on QM9.
Molecular dynamic simulations are important in computational physics, chemistry, material, and biology. Machine learning-based methods have shown strong abilities in predicting molecular energy and properties and are much faster than DFT calculations. Molecular energy is at least related to atoms, bonds, bond angles, torsion angles, and nonbonding atom pairs. Previous Transformer models only use atoms as inputs which lack explicit modeling of the aforementioned factors. To alleviate this limitation, we propose Moleformer, a novel Transformer architecture that takes nodes (atoms) and edges (bonds and nonbonding atom pairs) as inputs and models the interactions among them using rotational and translational invariant geometry-aware spatial encoding. Proposed spatial encoding calculates relative position information including distances and angles among nodes and edges. We benchmark Moleformer on OC20 and QM9 datasets, and our model achieves state-of-the-art on the initial state to relaxed energy prediction of OC20 and is very competitive in QM9 on predicting quantum chemical properties compared to other Transformer and Graph Neural Network (GNN) methods which proves the effectiveness of the proposed geometry-aware spatial encoding in Moleformer.