Transformers for scientific data: a pedagogical review for astronomers
It provides an accessible guide for astronomers and scientists to understand and apply transformers in their research, but it is incremental as it reviews existing knowledge without presenting new results.
This pedagogical review introduces transformers, a deep learning architecture, to scientists, covering the underlying mathematics, architecture, and applications in astronomy for time series and imaging data.
The deep learning architecture associated with ChatGPT and related generative AI products is known as transformers. Initially applied to Natural Language Processing, transformers and the self-attention mechanism they exploit have gained widespread interest across the natural sciences. The goal of this pedagogical and informal review is to introduce transformers to scientists. The review includes the mathematics underlying the attention mechanism, a description of the original transformer architecture, and a section on applications to time series and imaging data in astronomy. We include a Frequently Asked Questions section for readers who are curious about generative AI or interested in getting started with transformers for their research problem.