Jia Li

Jia Li, Ge Li, Yunfei Zhao et al. · pku

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

Wenxiang Cong, Ge Wang, Qingsong Yang et al.

Regularization methods are commonly used in X-ray CT image reconstruction. Different regularization methods reflect the characterization of different prior knowledge of images. In a recent work, a new regularization method called a low-dimensional manifold model (LDMM) is investigated to characterize the low-dimensional patch manifold structure of natural images, where the manifold dimensionality characterizes structural information of an image. In this paper, we propose a CT image reconstruction method based on the prior knowledge of the low-dimensional manifold of CT image. Using the clinical raw projection data from GE clinic, we conduct comparisons for the CT image reconstruction among the proposed method, the simultaneous algebraic reconstruction technique (SART) with the total variation (TV) regularization, and the filtered back projection (FBP) method. Results show that the proposed method can successfully recover structural details of an imaging object, and achieve higher spatial and contrast resolution of the reconstructed image than counterparts of FBP and SART with TV.