Low-resource domain adaptation while minimizing energy and hardware resource consumption
This work addresses energy efficiency and accessibility issues for research groups with limited infrastructure, though it is incremental as it applies existing techniques to a specific bottleneck.
The paper tackles the problem of high computational costs in domain adaptation for Large Language Models (LLMs) by evaluating numerical precision formats and data parallelization strategies, finding that these methods can reduce energy and hardware consumption while maintaining model accuracy in low-resource environments.
Training Large Language Models (LLMs) is costly in terms of energy, hardware, and annotated data, often resulting in a positionality rooted in predominant cultures and values (Santy et al., 2023). Domain adaptation has emerged as a promising strategy to better align models with diverse cultural and value contexts (Hershcovich et al., 2022), but its computational cost remains a significant barrier, particularly for research groups lacking access to large-scale infrastructure. In this paper, we evaluate how the use of different numerical precision formats and data parallelization strategies impacts both training speed (as a proxy to energy and hardware consumption) and model accuracy, with the goal of facilitating domain adaptation in low-resource environments. Our findings are relevant to any setting where energy efficiency, accessibility, or limited hardware availability are key concerns.