How Green are Neural Language Models? Analyzing Energy Consumption in Text Summarization Fine-tuning
It addresses the environmental impact of AI for NLP researchers and practitioners, but is incremental as it applies existing methods to measure energy consumption without proposing new solutions.
This study analyzed the energy consumption and performance trade-offs of fine-tuning three neural language models (T5-base, BART-base, and LLaMA-3-8B) for text summarization, finding that LLaMA-3-8B had the largest carbon footprint.
Artificial intelligence systems significantly impact the environment, particularly in natural language processing (NLP) tasks. These tasks often require extensive computational resources to train deep neural networks, including large-scale language models containing billions of parameters. This study analyzes the trade-offs between energy consumption and performance across three neural language models: two pre-trained models (T5-base and BART-base), and one large language model (LLaMA-3-8B). These models were fine-tuned for the text summarization task, focusing on generating research paper highlights that encapsulate the core themes of each paper. The carbon footprint associated with fine-tuning each model was measured, offering a comprehensive assessment of their environmental impact. It is observed that LLaMA-3-8B produces the largest carbon footprint among the three models. A wide range of evaluation metrics, including ROUGE, METEOR, MoverScore, BERTScore, and SciBERTScore, were employed to assess the performance of the models on the given task. This research underscores the importance of incorporating environmental considerations into the design and implementation of neural language models and calls for the advancement of energy-efficient AI methodologies.