LLavaCode: Compressed Code Representations for Retrieval-Augmented Code Generation
This addresses a critical limitation for interactive settings like IDEs by making code completion faster and more efficient, though it is an incremental improvement on existing retrieval-augmented methods.
The paper tackles the problem of slow inference in retrieval-augmented code generation by introducing LlavaCode, a framework that compresses code into compact representations, reducing context to single-token vectors. This approach achieves a 20-38% reduction in Time-to-First-Token for line completion tasks while improving generation quality metrics like EM and ES with negligible latency increase.
Retrieval-augmented generation has emerged as one of the most effective approaches for code completion, particularly when context from a surrounding repository is essential. However, incorporating context significantly extends sequence length, leading to slower inference - a critical limitation for interactive settings such as IDEs. In this work, we introduce LlavaCode, a framework that compresses code into compact, semantically rich representations interpretable by code LLM, enhancing generation quality while reducing the retrieved context to only a few compressed single-token vectors. Using a small projector module we can significantly increase the EM and ES metrics of coding model with negligible latency increase. Our experiments demonstrate that compressed context enables 20-38% reduction in Time-to-First-Token (TTFT) on line completion tasks compared to full-RAG pipelines.