A Survey on Memory-Efficient Transformer-Based Model Training in AI for Science
This is an incremental survey that addresses memory inefficiencies for researchers and practitioners using LLMs in scientific fields like biology and chemistry.
This survey tackles the problem of high memory demands in transformer-based large language models (LLMs) for scientific applications, reviewing memory-efficient pre-training techniques and demonstrating with AlphaFold 2 that tailored optimizations can reduce storage needs while maintaining accuracy.
Scientific research faces high costs and inefficiencies with traditional methods, but the rise of deep learning and large language models (LLMs) offers innovative solutions. This survey reviews transformer-based LLM applications across scientific fields such as biology, medicine, chemistry, and meteorology, underscoring their role in advancing research. However, the continuous expansion of model size has led to significant memory demands, hindering further development and application of LLMs for science. This survey systematically reviews and categorizes memory-efficient pre-training techniques for large-scale transformers, including algorithm-level, system-level, and hardware-software co-optimization. Using AlphaFold 2 as an example, we demonstrate how tailored memory optimization methods can reduce storage needs while preserving prediction accuracy. By bridging model efficiency and scientific application needs, we hope to provide insights for scalable and cost-effective LLM training in AI for science.