Toward Understanding Privileged Features Distillation in Learning-to-Rank
This work addresses the challenge of using expensive or unavailable features in ranking systems for recommendation platforms, though it is incremental as it builds on existing distillation methods.
The paper tackles the problem of leveraging privileged features in learning-to-rank, which are available during training but not at test time, and shows that privileged features distillation outperforms baselines on multiple datasets, including an industrial-scale one from Amazon. It reveals a non-monotonic effect where increasing predictive power of privileged features initially boosts but then harms student model performance due to high variance in teacher predictions.
In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive of "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. Privileged features distillation (PFD) refers to a natural idea: train a "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features. In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets. Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.