On the Fairness of Randomized Trials for Recommendation with Heterogeneous Demographics and Beyond
This addresses fairness and bias correction in recommendation systems, particularly for heterogeneous demographic groups, though it appears incremental as it builds on existing RCT methods.
The paper tackles bias in recommendation systems due to missing not at random data by analyzing fairness issues in randomized controlled trials and proposing a counterfactual robust risk minimization framework, with empirical results showing superiority in fairness and generalization on synthetic and real-world datasets.
Observed events in recommendation are consequence of the decisions made by a policy, thus they are usually selectively labeled, namely the data are Missing Not At Random (MNAR), which often causes large bias to the estimate of true outcomes risk. A general approach to correct MNAR bias is performing small Randomized Controlled Trials (RCTs), where an additional uniform policy is employed to randomly assign items to each user. In this work, we concentrate on the fairness of RCTs under both homogeneous and heterogeneous demographics, especially analyzing the bias for the least favorable group on the latter setting. Considering RCTs' limitations, we propose a novel Counterfactual Robust Risk Minimization (CRRM) framework, which is totally free of expensive RCTs, and derive its theoretical generalization error bound. At last, empirical experiments are performed on synthetic tasks and real-world data sets, substantiating our method's superiority both in fairness and generalization.