Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia
This addresses the problem of assessing LLMs for practical professional demands in Indonesia, though it is incremental as it extends evaluation to a new domain-specific dataset.
The authors tackled the gap in evaluating large language models on real-world professional knowledge by introducing IndoCareer, a dataset of 8,834 multiple-choice questions from Indonesian vocational exams across six sectors, and found that models struggle most in fields with strong local contexts like insurance and finance.
While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.