Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval
This addresses a bottleneck in semantic indexing for recommendation and retrieval systems, offering an incremental improvement over existing methods.
The paper tackles the problem of semantic ID conflicts in LLM-based generative recommendation and retrieval, where similar items get identical IDs, by proposing purely semantic indexing to generate unique IDs without non-semantic tokens, resulting in improved overall and cold-start performance in experiments on sequential recommendation, product search, and document retrieval.
Semantic identifiers (IDs) have proven effective in adapting large language models for generative recommendation and retrieval. However, existing methods often suffer from semantic ID conflicts, where semantically similar documents (or items) are assigned identical IDs. A common strategy to avoid conflicts is to append a non-semantic token to distinguish them, which introduces randomness and expands the search space, therefore hurting performance. In this paper, we propose purely semantic indexing to generate unique, semantic-preserving IDs without appending non-semantic tokens. We enable unique ID assignment by relaxing the strict nearest-centroid selection and introduce two model-agnostic algorithms: exhaustive candidate matching (ECM) and recursive residual searching (RRS). Extensive experiments on sequential recommendation, product search, and document retrieval tasks demonstrate that our methods improve both overall and cold-start performance, highlighting the effectiveness of ensuring ID uniqueness.