OpenStaxQA: A multilingual dataset based on open-source college textbooks
This provides a domain-specific benchmark for college-level educational applications, but it is incremental as it adapts existing methods to new data.
The authors introduced OpenStaxQA, a multilingual dataset derived from 43 open-source college textbooks in English, Spanish, and Polish, and used it to finetune and evaluate large language models with about 7 billion parameters via quantized low rank adapters, while also testing zero-shot performance on the AI2 reasoning challenge.
We present OpenStaxQA, an evaluation benchmark specific to college-level educational applications based on 43 open-source college textbooks in English, Spanish, and Polish, available under a permissive Creative Commons license. We finetune and evaluate large language models (LLMs) with approximately 7 billion parameters on this dataset using quantized low rank adapters (QLoRa). Additionally we also perform a zero-shot evaluation on the AI2 reasoning challenge dev dataset in order to check if OpenStaxQA can lead to an improved performance on other tasks. We also discuss broader impacts relevant to datasets such as OpenStaxQA.