LLM-as-a-Judge: Toward World Models for Slate Recommendation Systems
This work addresses a key problem in slate recommendation systems for improving user modeling, but it appears incremental as it builds on existing LLM capabilities without introducing a new paradigm.
The paper tackles the challenge of modeling user preferences across domains in slate recommendation systems by investigating how Large Language Models (LLMs) can act as world models through pairwise reasoning over slates. The results reveal relationships between task performance and properties of the preference function captured by LLMs, highlighting their potential in recommender systems.
Modeling user preferences across domains remains a key challenge in slate recommendation (i.e. recommending an ordered sequence of items) research. We investigate how Large Language Models (LLM) can effectively act as world models of user preferences through pairwise reasoning over slates. We conduct an empirical study involving several LLMs on three tasks spanning different datasets. Our results reveal relationships between task performance and properties of the preference function captured by LLMs, hinting towards areas for improvement and highlighting the potential of LLMs as world models in recommender systems.