Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation
This work improves generative recommendation systems by better aligning tokenization and modeling stages, which is important for platforms needing accurate and context-aware item suggestions.
The paper tackles the problem of objective misalignment in two-stage generative recommenders, where pretrained tokenizers and LLMs have different optimization goals, leading to suboptimal tokenization and loss of pretrained semantics. The proposed DECOR framework addresses this by learning decomposed contextual token representations, resulting in consistent outperformance of state-of-the-art baselines on three real-world datasets.
Recent advances in generative recommenders adopt a two-stage paradigm: items are first tokenized into semantic IDs using a pretrained tokenizer, and then large language models (LLMs) are trained to generate the next item via sequence-to-sequence modeling. However, these two stages are optimized for different objectives: semantic reconstruction during tokenizer pretraining versus user interaction modeling during recommender training. This objective misalignment leads to two key limitations: (i) suboptimal static tokenization, where fixed token assignments fail to reflect diverse usage contexts; and (ii) discarded pretrained semantics, where pretrained knowledge - typically from language model embeddings - is overwritten during recommender training on user interactions. To address these limitations, we propose to learn DEcomposed COntextual Token Representations (DECOR), a unified framework that preserves pretrained semantics while enhancing the adaptability of token embeddings. DECOR introduces contextualized token composition to refine token embeddings based on user interaction context, and decomposed embedding fusion that integrates pretrained codebook embeddings with newly learned collaborative embeddings. Experiments on three real-world datasets demonstrate that DECOR consistently outperforms state-of-the-art baselines in recommendation performance. Our code will be made available upon publication.