Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done!
This work addresses the need for reliable in-context personalization in LLMs for summarization, which is crucial for users requiring tailored content, but it is incremental as it builds on existing measures like EGISES.
The paper tackles the problem of evaluating whether large language models (LLMs) can truly perform in-context personalized summarization by discerning user preferences, and finds that 15 out of 17 state-of-the-art models degrade in performance (by 1.6% to 3.6%) when tested with richer prompts, indicating a lack of true personalization capability.
Large Language Models (LLMs) have succeeded considerably in In-Context-Learning (ICL) based summarization. However, saliency is subject to the users' specific preference histories. Hence, we need reliable In-Context Personalization Learning (ICPL) capabilities within such LLMs. For any arbitrary LLM to exhibit ICPL, it needs to have the ability to discern contrast in user profiles. A recent study proposed a measure for degree-of-personalization called EGISES for the first time. EGISES measures a model's responsiveness to user profile differences. However, it cannot test if a model utilizes all three types of cues provided in ICPL prompts: (i) example summaries, (ii) user's reading histories, and (iii) contrast in user profiles. To address this, we propose the iCOPERNICUS framework, a novel In-COntext PERsonalization learNIng sCrUtiny of Summarization capability in LLMs that uses EGISES as a comparative measure. As a case-study, we evaluate 17 state-of-the-art LLMs based on their reported ICL performances and observe that 15 models' ICPL degrades (min: 1.6%; max: 3.6%) when probed with richer prompts, thereby showing lack of true ICPL.