LaMP-QA: A Benchmark for Personalized Long-form Question Answering
This addresses the problem of personalization in user-centric QA systems by providing a new benchmark, though it is incremental as it builds on existing QA and personalization research.
The paper tackles the lack of resources for personalized question answering by introducing LaMP-QA, a benchmark for evaluating personalized long-form answer generation, showing that incorporating personalized context leads to up to 39% performance improvements.
Personalization is essential for question answering systems that are user-centric. Despite its importance, personalization in answer generation has been relatively underexplored. This is mainly due to lack of resources for training and evaluating personalized question answering systems. We address this gap by introducing LaMP-QA -- a benchmark designed for evaluating personalized long-form answer generation. The benchmark covers questions from three major categories: (1) Arts & Entertainment, (2) Lifestyle & Personal Development, and (3) Society & Culture, encompassing over 45 subcategories in total. To assess the quality and potential impact of the LaMP-QA benchmark for personalized question answering, we conduct comprehensive human and automatic evaluations, to compare multiple evaluation strategies for evaluating generated personalized responses and measure their alignment with human preferences. Furthermore, we benchmark a number of non-personalized and personalized approaches based on open-source and proprietary large language models. Our results show that incorporating the personalized context provided leads to up to 39% performance improvements. The benchmark is publicly released to support future research in this area.