Foundation models in brief: A historical, socio-technical focus
It offers a concise overview for researchers and practitioners interested in the evolution and societal impacts of AI, but is incremental as it synthesizes existing knowledge without new empirical results.
This paper provides a short introduction to foundation models, distinguishing them from prior deep learning models and discussing their historical context and socio-technical implications, such as power shifts due to homogenization.
Foundation models can be disruptive for future AI development by scaling up deep learning in terms of model size and training data's breadth and size. These models achieve state-of-the-art performance (often through further adaptation) on a variety of tasks in domains such as natural language processing and computer vision. Foundational models exhibit a novel {emergent behavior}: {In-context learning} enables users to provide a query and a few examples from which a model derives an answer without being trained on such queries. Additionally, {homogenization} of models might replace a myriad of task-specific models with fewer very large models controlled by few corporations leading to a shift in power and control over AI. This paper provides a short introduction to foundation models. It contributes by crafting a crisp distinction between foundation models and prior deep learning models, providing a history of machine learning leading to foundation models, elaborating more on socio-technical aspects, i.e., organizational issues and end-user interaction, and a discussion of future research.