CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
For software engineers and AI researchers, this work addresses the overlooked visual and structural aspects in code retrieval, enabling more comprehensive search and improved RAG-based code generation.
The paper introduces MMCoIR, the first benchmark for multimodal code IR across five visual domains, and proposes CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images. CodeMMR outperforms baselines by an average of 10 points on nDCG@10 and improves code generation fidelity in RAG.
Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.