SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair
This addresses the problem of ensuring functional correctness in smart contract code completion for blockchain developers, though it is incremental as it builds on existing code completion models.
The authors tackled the lack of benchmarks for evaluating functional correctness in Solidity smart contract code completion by introducing SolBench, a dataset of 4,178 functions from deployed contracts, and found that retrieval-augmented repair techniques improved correctness while reducing computational costs.
Smart contracts are crucial programs on blockchains, and their immutability post-deployment makes functional correctness vital. Despite progress in code completion models, benchmarks for Solidity, the primary smart contract language, are lacking. Existing metrics like BLEU do not adequately assess the functional correctness of generated smart contracts. To fill this gap, we introduce SolBench, a benchmark for evaluating the functional correctness of Solidity smart contracts generated by code completion models. SolBench includes 4,178 functions from 1,155 Ethereum-deployed contracts. Testing advanced models revealed challenges in generating correct code without context, as Solidity functions rely on context-defined variables and interfaces. To address this, we propose a Retrieval-Augmented Code Repair framework. In this framework, an executor verifies functional correctness, and if necessary, an LLM repairs the code using retrieved snippets informed by executor traces. We conduct a comprehensive evaluation of both closed-source and open-source LLMs across various model sizes and series to assess their performance in smart contract completion. The results show that code repair and retrieval techniques effectively enhance the correctness of smart contract completion while reducing computational costs.