Rank4Gen: RAG-Preference-Aligned Document Set Selection and Ranking
For RAG systems, this work addresses the underexplored problem of generator-specific document set selection and ordering, improving downstream QA quality.
The paper identifies that RAG document ranking should consider document composition and ordering, which vary across generators, and proposes Rank4Gen, a generator-aware ranker trained on a new preference-aligned dataset PRISM, achieving up to +2.08 F1 gains over baselines.
In the RAG paradigm, document ranking determines the evidence available to downstream generators. Through controlled analysis, we identify two phenomena underexplored by existing rankers: (i) downstream response quality depends not only on relevance but also on the composition and ordering of selected documents, and (ii) such preferences differ systematically across generators. However, existing rankers are trained purely on query--document relevance, leaving both phenomena unmodeled. To close this gap, we construct \textbf{PRISM}, a bilingual preference-aligned dataset built through a four-stage pipeline that compresses the combinatorial subset-and-ordering space by roughly four orders of magnitude and produces response-quality preference supervision conditioned on seven downstream generators. On a 13k-query subset of PRISM, we train \textbf{Rank4Gen}, a generator-aware ranker that performs joint document set selection and ordering. Experiments on five challenging RAG benchmarks show that Rank4Gen improves downstream QA quality on most evaluated generators, with per-generator F1 gains of up to $+2.08$ over the strongest set-selection baseline. Code is available at https://github.com/JOHNNY-fans/Rank4Gen.