Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino
This addresses the issue of cultural representation in LLMs for Filipino users, but it is incremental as it focuses on a specific domain.
The paper tackles the problem of multilingual LLMs lacking culturally appropriate responses for Filipino users by introducing Kalahi, a handcrafted evaluation suite of 150 prompts, resulting in the best model achieving only 46.0% accuracy compared to native Filipino performance of 89.10%.
Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs.