Interactive Episodic Memory with User Feedback
This work addresses the practical limitation of ambiguous queries in episodic memory retrieval by enabling interactive user feedback, which is crucial for real-world applications like personal assistants.
The paper introduces the Episodic Memory with Questions and Feedback (EM-QnF) task, where users can provide feedback to refine predictions in episodic memory with natural language queries. The proposed Feedback ALignment Module (FALM) achieves significant improvements over state-of-the-art on three benchmarks and is competitive with commercial models while remaining efficient.
In episodic memory with natural language queries (EM-NLQ), a user may ask a question (e.g., "Where did I place the mug?") that requires searching a long egocentric video, captured from the user's perspective, to find the moment that answers it. However, queries can be ambiguous or incomplete, leading to incorrect responses. Current methods ignore this key aspect and address EM-NLQ in a one-shot setup, limiting their applicability in real-world scenarios. In this work, we address this gap and introduce the Episodic Memory with Questions and Feedback task (EM-QnF). Here, the user can provide feedback on the model's initial prediction or add more information (e.g., "Before this. I'm looking for the big blue mug not the white one"), helping the model refine its predictions interactively. To this end, we collect datasets for feedback-based interaction and propose a lightweight training scheme that avoids expensive sequential optimization. We also introduce a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively. Our approach significantly improves over the state of the art on three challenging benchmarks and is better than or competitive with commercial large vision-language models while remaining efficient. Evaluation with human-generated feedback shows that it generalizes well to real-world scenarios.