Can ChatGPT Learn My Life From a Week of First-Person Video?
This addresses the problem of personal AI learning from wearable data for users, but it is incremental as it builds on existing models and highlights limitations like hallucination.
The study tested whether foundation models like GPT-4o can learn personal details from first-person video data, finding that while they correctly deduced basic information such as location, occupation, and pet ownership, they also suffered from hallucinations like making up names for people in the footage.
Motivated by recent improvements in generative AI and wearable camera devices (e.g. smart glasses and AI-enabled pins), I investigate the ability of foundation models to learn about the wearer's personal life through first-person camera data. To test this, I wore a camera headset for 54 hours over the course of a week, generated summaries of various lengths (e.g. minute-long, hour-long, and day-long summaries), and fine-tuned both GPT-4o and GPT-4o-mini on the resulting summary hierarchy. By querying the fine-tuned models, we are able to learn what the models learned about me. The results are mixed: Both models learned basic information about me (e.g. approximate age, gender). Moreover, GPT-4o correctly deduced that I live in Pittsburgh, am a PhD student at CMU, am right-handed, and have a pet cat. However, both models also suffered from hallucination and would make up names for the individuals present in the video footage of my life.