Challenge on Optimization of Context Collection for Code Completion
This addresses the problem of optimizing context collection for code completion in large code bases, which is incremental as it builds on existing methods through a competition framework.
The paper describes a challenge organized by JetBrains and Mistral AI at ASE 2025, where participants developed efficient context collection mechanisms from source code repositories to improve fill-in-the-middle code completions for Python and Kotlin, using a large dataset of real-world code and evaluating submissions with the chrF metric on state-of-the-art neural models.
The rapid advancement of workflows and methods for software engineering using AI emphasizes the need for a systematic evaluation and analysis of their ability to leverage information from entire projects, particularly in large code bases. In this challenge on optimization of context collection for code completion, organized by JetBrains in collaboration with Mistral AI as part of the ASE 2025 conference, participants developed efficient mechanisms for collecting context from source code repositories to improve fill-in-the-middle code completions for Python and Kotlin. We constructed a large dataset of real-world code in these two programming languages using permissively licensed open-source projects. The submissions were evaluated based on their ability to maximize completion quality for multiple state-of-the-art neural models using the chrF metric. During the public phase of the competition, nineteen teams submitted solutions to the Python track and eight teams submitted solutions to the Kotlin track. In the private phase, six teams competed, of which five submitted papers to the workshop.