Heuristic-Based Weak Learning for Automated Decision-Making
This addresses the barrier to participation in algorithm design for affected users, offering a more practical alternative to high-volume labeling, though it appears incremental as it builds on prior aggregation techniques.
The paper tackled the problem of costly manual labeling for reconciling conflicting user preferences in automated decision-making by proposing a weak learning approach that collects ranked heuristics from users, showing it agrees with participants' pairwise choices nearly as often as fully supervised methods in empirical use cases.
Machine learning systems impact many stakeholders and groups of users, often disparately. Prior studies have reconciled conflicting user preferences by aggregating a high volume of manually labeled pairwise comparisons, but this technique may be costly or impractical. How can we lower the barrier to participation in algorithm design? Instead of creating a simplified labeling task for a crowd, we suggest collecting ranked decision-making heuristics from a focused sample of affected users. With empirical data from two use cases, we show that our weak learning approach, which requires little to no manual labeling, agrees with participants' pairwise choices nearly as often as fully supervised approaches.