TST$^\mathrm{R}$: Target Similarity Tuning Meets the Real World
This work addresses the challenge of adapting TST for real-world applications in code generation, offering incremental improvements to enhance efficiency and practicality.
The paper tackles the problem of improving target similarity tuning (TST) for selecting relevant examples in natural language to code generation using large language models, by proposing methods such as using larger model embeddings, efficient training example selection, and a ranking-based evaluation, resulting in reduced sensitivity to language distribution and more flexible synthetic generation.
Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs) to improve performance. Its goal is to adapt a sentence embedding model to have the similarity between two NL inputs match the similarity between their associated code outputs. In this paper, we propose different methods to apply and improve TST in the real world. First, we replace the sentence transformer with embeddings from a larger model, which reduces sensitivity to the language distribution and thus provides more flexibility in synthetic generation of examples, and we train a tiny model that transforms these embeddings to a space where embedding similarity matches code similarity, which allows the model to remain a black box and only requires a few matrix multiplications at inference time. Second, we show how to efficiently select a smaller number of training examples to train the TST model. Third, we introduce a ranking-based evaluation for TST that does not require end-to-end code generation experiments, which can be expensive to perform.