Recording First-person Experiences to Build a New Type of Foundation Model
This addresses the need for richer, more authentic data sources for AI foundation models, potentially improving applications like personal assistants and chatbots, but it is incremental as it builds on existing foundation model concepts with new data types.
The authors tackled the problem of limited and superficial data for training foundation models by developing a recording rig that captures first-person experiences, including visual, auditory, and physiological data like skin conductance and EEG, to enable more accurate replication of human behavior. They propose using this data to create a new type of foundation model with applications in recommendation, personal assistance, and other domains, though the work is preliminary and focused on data collection and startup funding.
Foundation models have had a big impact in recent years and billions of dollars are being invested in them in the current AI boom. The more popular ones, such as Chat-GPT, are trained on large amounts of Internet data. However, it is becoming apparent that this data is likely to be exhausted soon, and technology companies are looking for new sources of data to train the next generation of foundation models. Reinforcement learning, RAG, prompt engineering and cognitive modelling are often used to fine-tune and augment the behaviour of foundation models. These techniques have been used to replicate people, such as Caryn Marjorie. These chatbots are not based on people's actual emotional and physiological responses to their environment, so they are, at best, a surface-level approximation to the characters they are imitating. To address these issues, we have developed a recording rig that captures what the wearer is seeing and hearing as well as their skin conductance (GSR), facial expression and brain state (14 channel EEG). AI algorithms are used to process this data into a rich picture of the environment and internal states of the subject. Foundation models trained on this data could replicate human behaviour much more accurately than the personality models that have been developed so far. This type of model has many potential applications, including recommendation, personal assistance, GAN systems, dating and recruitment. This paper gives some background to this work and describes the recording rig and preliminary tests of its functionality. It then suggests how a new type of foundation model could be created from the data captured by the rig and outlines some applications. Data gathering and model training are expensive, so we are currently working on the launch of a start-up that could raise funds for the next stage of the project.