Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs
This addresses the bottleneck of data annotation for AI/ML developers by introducing a collaborative human-LLM approach, though it appears incremental as it builds on existing annotation methods with LLM assistance.
The paper tackles the problem of slow, labor-intensive, and inconsistent AI training data annotation by proposing the Model-in-the-Loop (MILO) framework, which integrates large language models (LLMs) with human annotators; empirical studies show it reduces handling time, improves data quality, and enhances annotator experiences.
The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality. We propose the Model-in-the-Loop (MILO) framework, which integrates AI/ML models into the annotation process. Our research introduces a collaborative paradigm that leverages the strengths of both professional human annotators and large language models (LLMs). By employing LLMs as pre-annotation and real-time assistants, and judges on annotator responses, MILO enables effective interaction patterns between human annotators and LLMs. Three empirical studies on multimodal data annotation demonstrate MILO's efficacy in reducing handling time, improving data quality, and enhancing annotator experiences. We also introduce quality rubrics for flexible evaluation and fine-grained feedback on open-ended annotations. The MILO framework has implications for accelerating AI/ML development, reducing reliance on human annotation alone, and promoting better alignment between human and machine values.