Should attention be all we need? The epistemic and ethical implications of unification in machine learning
This work critiques a foundational trend in ML for researchers and practitioners, highlighting potential negative impacts on diversity and ethics, but it is incremental as it builds on existing philosophical and ethical discussions without introducing new methods or data.
The paper examines the epistemic and ethical implications of unifying machine learning around transformer architectures, arguing that such unification introduces risks like methodological diversity loss and power centralization, without presenting new empirical results or concrete numbers.
"Attention is all you need" has become a fundamental precept in machine learning research. Originally designed for machine translation, transformers and the attention mechanisms that underpin them now find success across many problem domains. With the apparent domain-agnostic success of transformers, many researchers are excited that similar model architectures can be successfully deployed across diverse applications in vision, language and beyond. We consider the benefits and risks of these waves of unification on both epistemic and ethical fronts. On the epistemic side, we argue that many of the arguments in favor of unification in the natural sciences fail to transfer over to the machine learning case, or transfer over only under assumptions that might not hold. Unification also introduces epistemic risks related to portability, path dependency, methodological diversity, and increased black-boxing. On the ethical side, we discuss risks emerging from epistemic concerns, further marginalizing underrepresented perspectives, the centralization of power, and having fewer models across more domains of application