Biquality Learning: a Framework to Design Algorithms Dealing with Closed-Set Distribution Shifts
This addresses the problem of robust algorithm design for researchers and practitioners dealing with complex distribution shifts in real-world data, though it is incremental as it builds on existing literature.
The paper tackles the challenge of training machine learning models under weak supervision and dataset shifts by proposing the biquality learning framework, which uses trusted and untrusted datasets to design algorithms that handle distributional shifts, with methods inspired by label noise and covariate shift literature.
Training machine learning models from data with weak supervision and dataset shifts is still challenging. Designing algorithms when these two situations arise has not been explored much, and existing algorithms cannot always handle the most complex distributional shifts. We think the biquality data setup is a suitable framework for designing such algorithms. Biquality Learning assumes that two datasets are available at training time: a trusted dataset sampled from the distribution of interest and the untrusted dataset with dataset shifts and weaknesses of supervision (aka distribution shifts). The trusted and untrusted datasets available at training time make designing algorithms dealing with any distribution shifts possible. We propose two methods, one inspired by the label noise literature and another by the covariate shift literature for biquality learning. We experiment with two novel methods to synthetically introduce concept drift and class-conditional shifts in real-world datasets across many of them. We opened some discussions and assessed that developing biquality learning algorithms robust to distributional changes remains an interesting problem for future research.