CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility
This addresses a gap in evaluating LLMs for European Portuguese users, though it is incremental as it adapts existing leaderboard concepts to a specific language variant.
The authors tackled the lack of a dedicated leaderboard for evaluating Open Large Language Models in European Portuguese by developing CLARIN-PT-LDB, which includes novel benchmarks for model safeguards and cultural alignment, making it publicly available online.
This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.