CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains
This provides a standardized framework for researchers to assess and compare Chinese chat models, addressing a gap in NLG evaluation, though it is incremental as it adapts existing benchmark concepts to a specific language and domain.
The authors tackled the lack of standardized evaluation benchmarks for Chinese generative chat models by introducing the CGCE benchmark, which includes 200 general and 150 financial domain questions and uses manual scoring for factors like accuracy and coherence.
Generative chat models, such as ChatGPT and GPT-4, have revolutionized natural language generation (NLG) by incorporating instructions and human feedback to achieve significant performance improvements. However, the lack of standardized evaluation benchmarks for chat models, particularly for Chinese and domain-specific models, hinders their assessment and progress. To address this gap, we introduce the Chinese Generative Chat Evaluation (CGCE) benchmark, focusing on general and financial domains. The CGCE benchmark encompasses diverse tasks, including 200 questions in the general domain and 150 specific professional questions in the financial domain. Manual scoring evaluates factors such as accuracy, coherence, expression clarity, and completeness. The CGCE benchmark provides researchers with a standardized framework to assess and compare Chinese generative chat models, fostering advancements in NLG research.