Understanding QA generation: Extracting Parametric and Contextual Knowledge with CQA for Low Resource Bangla Language
This addresses the problem of understanding knowledge sources in QA models for low-resource languages like Bangla, which is incremental as it builds on existing QA and counterfactual reasoning methods.
The researchers tackled the challenge of analyzing whether QA models for low-resource Bangla rely on pre-encoded or contextual knowledge by introducing BanglaCQA, the first counterfactual QA dataset in Bangla, and found that Chain-of-Thought prompting is particularly effective for extracting parametric knowledge in counterfactual scenarios with decoder-only LLMs.
Question-Answering (QA) models for low-resource languages like Bangla face challenges due to limited annotated data and linguistic complexity. A key issue is determining whether models rely more on pre-encoded (parametric) knowledge or contextual input during answer generation, as existing Bangla QA datasets lack the structure required for such analysis. We introduce BanglaCQA, the first Counterfactual QA dataset in Bangla, by extending a Bangla dataset while integrating counterfactual passages and answerability annotations. In addition, we propose fine-tuned pipelines for encoder-decoder language-specific and multilingual baseline models, and prompting-based pipelines for decoder-only LLMs to disentangle parametric and contextual knowledge in both factual and counterfactual scenarios. Furthermore, we apply LLM-based and human evaluation techniques that measure answer quality based on semantic similarity. We also present a detailed analysis of how models perform across different QA settings in low-resource languages, and show that Chain-of-Thought (CoT) prompting reveals a uniquely effective mechanism for extracting parametric knowledge in counterfactual scenarios, particularly in decoder-only LLMs. Our work not only introduces a novel framework for analyzing knowledge sources in Bangla QA but also uncovers critical findings that open up broader directions for counterfactual reasoning in low-resource language settings.