CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
This work addresses a gap in benchmarking LLMs for low-resource general-purpose languages, which is incremental as it builds on existing benchmarks like HumanEval and ClassEval.
The paper tackles the problem of LLMs struggling with low-resource general-purpose programming languages by introducing CangjieBench, a benchmark for Cangjie, and finds that Syntax-Constrained Generation offers the best trade-off in accuracy and cost, while Agent achieves state-of-the-art accuracy but with high token consumption.
Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.