ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
This addresses the challenge of reliable code generation for developers by providing a more robust selection method, though it is incremental as it builds on existing code selection algorithms.
The paper tackles the problem of selecting the correct program from multiple LLM-generated options by introducing ExPairT-LLM, an exact learning algorithm that uses pairwise queries to an LLM oracle, resulting in an average pass@1 improvement of +13.0% over state-of-the-art methods.
Despite recent advances in LLMs, the task of code generation is still challenging. To cope, code selection algorithms select the best program from multiple programs generated by an LLM. However, existing algorithms can fail to identify the correct program, either because they can misidentify nonequivalent programs or because they rely on an LLM and assume it always correctly determines the output for every input. We present ExPairT-LLM, an exact learning algorithm for code selection that selects a program by posing to an LLM oracle two new types of queries: pairwise membership and pairwise equivalence. These queries are simpler for LLMs and enable ExPairT-LLM to identify the correct program through a tournament, which is robust to some LLM mistakes. We evaluate ExPairT-LLM on four popular code datasets. Its pass@1 (success rate) outperforms the state-of-the-art code selection algorithm on average by +13.0% and up to +27.1%. It also improves the pass@1 of LLMs performing complex reasoning by +24.0%.