AIANO: Enhancing Information Retrieval with AI-Augmented Annotation
This work addresses the problem of complex and inefficient dataset annotation for researchers and practitioners in information retrieval, representing an incremental improvement over existing tools.
The paper tackled the inefficiency of creating high-quality information retrieval datasets by developing AIANO, an AI-augmented annotation tool that integrates human expertise with LLM assistance, resulting in nearly doubled annotation speed and improved retrieval accuracy in a user study with 15 participants.
The rise of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has rapidly increased the need for high-quality, curated information retrieval datasets. These datasets, however, are currently created with off-the-shelf annotation tools that make the annotation process complex and inefficient. To streamline this process, we developed a specialized annotation tool - AIANO. By adopting an AI-augmented annotation workflow that tightly integrates human expertise with LLM assistance, AIANO enables annotators to leverage AI suggestions while retaining full control over annotation decisions. In a within-subject user study ($n = 15$), participants created question-answering datasets using both a baseline tool and AIANO. AIANO nearly doubled annotation speed compared to the baseline while being easier to use and improving retrieval accuracy. These results demonstrate that AIANO's AI-augmented approach accelerates and enhances dataset creation for information retrieval tasks, advancing annotation capabilities in retrieval-intensive domains.