Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization
This work addresses the problem of personalization in embodied agents for users, but it is incremental as it builds on existing memory utilization approaches to improve specific bottlenecks.
The study tackled the challenge of enabling LLM-powered embodied agents to provide personalized assistance by leveraging user-specific knowledge from past interactions, focusing on memory utilization for object semantics and user patterns. The result was the development of a hierarchical knowledge graph-based user-profile memory module that achieved substantial improvements on both single and joint-memory tasks.
LLM-powered embodied agents have shown success on conventional object-rearrangement tasks, but providing personalized assistance that leverages user-specific knowledge from past interactions presents new challenges. We investigate these challenges through the lens of agents' memory utilization along two critical dimensions: object semantics (identifying objects based on personal meaning) and user patterns (recalling sequences from behavioral routines). To assess these capabilities, we construct MEMENTO, an end-to-end two-stage evaluation framework comprising single-memory and joint-memory tasks. Our experiments reveal that current agents can recall simple object semantics but struggle to apply sequential user patterns to planning. Through in-depth analysis, we identify two critical bottlenecks: information overload and coordination failures when handling multiple memories. Based on these findings, we explore memory architectural approaches to address these challenges. Given our observation that episodic memory provides both personalized knowledge and in-context learning benefits, we design a hierarchical knowledge graph-based user-profile memory module that separately manages personalized knowledge, achieving substantial improvements on both single and joint-memory tasks. Project website: https://connoriginal.github.io/MEMENTO