When does Active Learning Work?
This work addresses a fundamental problem in machine learning for researchers and practitioners by providing a detailed methodology to assess when AL is beneficial, though it appears incremental as it focuses on evaluation rather than new methods.
The paper investigates the conditions under which Active Learning (AL) improves classifier performance when labels are scarce, presenting a comprehensive experimental simulation study across various tasks and classifiers to quantify its effectiveness.
Active Learning (AL) methods seek to improve classifier performance when labels are expensive or scarce. We consider two central questions: Where does AL work? How much does it help? To address these questions, a comprehensive experimental simulation study of Active Learning is presented. We consider a variety of tasks, classifiers and other AL factors, to present a broad exploration of AL performance in various settings. A precise way to quantify performance is needed in order to know when AL works. Thus we also present a detailed methodology for tackling the complexities of assessing AL performance in the context of this experimental study.