Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction
This work provides insights for researchers in computational biology and machine learning by identifying optimal training conditions for transformers in protein structure prediction, though it is incremental as it builds on existing methods.
The study tackled the problem of understanding why transformers excel in protein structure prediction by analyzing their loss landscape using statistical mechanics, finding that intermediate temperatures yield optimal learning without first-order transitions and that attention matrices better predict protein contact maps at higher temperatures.
We investigate the parameter space of transformer models trained on protein sequence data using a statistical mechanics framework, sampling the loss landscape at varying temperatures by Langevin dynamics to characterize the low-loss manifold and understand the mechanisms underlying the superior performance of transformers in protein structure prediction. We find that, at variance with feedforward networks, the lack of a first--order--like transition in the loss of the transformer produces a range of intermediate temperatures with good learning properties. We show that the parameters of most layers are highly conserved at these temperatures if the dimension of the embedding is optimal, and we provide an operative way to find this dimension. Finally, we show that the attention matrix is more predictive of the contact maps of the protein at higher temperatures and for higher dimensions of the embedding than those optimal for learning.