Evaluation of large language models using an Indian language LGBTI+ lexicon
This work addresses the need for domain-specific evaluation of LLMs to ensure responsible AI behavior for LGBTI+ communities, particularly in Indian languages, though it is incremental as it extends existing lexicon-based methods to a new domain.
The paper tackled the problem of evaluating large language models (LLMs) for responsible behavior in the LGBTI+ context using an Indian language lexicon, finding that three tested LLMs failed to detect underlying hateful content and highlighting limitations in machine translation for non-English language understanding.
Large language models (LLMs) are typically evaluated on the basis of task-based benchmarks such as MMLU. Such benchmarks do not examine responsible behaviour of LLMs in specific contexts. This is particularly true in the LGBTI+ context where social stereotypes may result in variation in LGBTI+ terminology. Therefore, domain-specific lexicons or dictionaries may be useful as a representative list of words against which the LLM's behaviour needs to be evaluated. This paper presents a methodology for evaluation of LLMs using an LGBTI+ lexicon in Indian languages. The methodology consists of four steps: formulating NLP tasks relevant to the expected behaviour, creating prompts that test LLMs, using the LLMs to obtain the output and, finally, manually evaluating the results. Our qualitative analysis shows that the three LLMs we experiment on are unable to detect underlying hateful content. Similarly, we observe limitations in using machine translation as means to evaluate natural language understanding in languages other than English. The methodology presented in this paper can be useful for LGBTI+ lexicons in other languages as well as other domain-specific lexicons. The work done in this paper opens avenues for responsible behaviour of LLMs, as demonstrated in the context of prevalent social perception of the LGBTI+ community.