Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison
Provides a diagnostic testbed for evaluating how AI agents handle conflicting and incomplete evidence from multiple personal memory sources, a key challenge for emerging personal AI agents.
The paper introduces a benchmark for selective QA over conflicting multi-source personal memory, with 34,560 instances across 18 question templates and 8 reasoning types. The best trained fusion resolver achieves 80.3% accuracy (85.3% selective accuracy at 78.3% coverage), outperforming the strongest LLM baseline at 70.0% accuracy (71.0% selective accuracy at 95.4% coverage).
Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method's conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse.