Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
This addresses the challenge of handling noisy and ambiguous data in computational biology and chemistry, offering a probabilistic approach for more accurate predictions and designs.
The paper tackles the problem of modeling ambiguous data in natural processes like RNA folding and molecule design by proposing a hierarchical latent distribution to enhance the Transformer, resulting in state-of-the-art performance on RNA folding and improved generative capabilities for molecule design.
Our world is ambiguous and this is reflected in the data we use to train our algorithms. This is particularly true when we try to model natural processes where collected data is affected by noisy measurements and differences in measurement techniques. Sometimes, the process itself is ambiguous, such as in the case of RNA folding, where the same nucleotide sequence can fold into different structures. This suggests that a predictive model should have similar probabilistic characteristics to match the data it models. Therefore, we propose a hierarchical latent distribution to enhance one of the most successful deep learning models, the Transformer, to accommodate ambiguities and data distributions. We show the benefits of our approach (1) on a synthetic task that captures the ability to learn a hidden data distribution, (2) with state-of-the-art results in RNA folding that reveal advantages on highly ambiguous data, and (3) demonstrating its generative capabilities on property-based molecule design by implicitly learning the underlying distributions and outperforming existing work.