Zero-Shot Commonsense Validation and Reasoning with Large Language Models: An Evaluation on SemEval-2020 Task 4 Dataset
This work addresses the problem of assessing commonsense reasoning in AI for researchers and practitioners, but it is incremental as it primarily benchmarks existing models on a known dataset.
This study evaluated large language models (LLMs) on the SemEval-2020 Task 4 dataset for commonsense validation and explanation using zero-shot prompting, finding that larger models like LLaMA3-70B achieved high accuracy (e.g., 98.40% in validation) but struggled with explanation tasks (e.g., 93.40% in explanation).
This study evaluates the performance of Large Language Models (LLMs) on SemEval-2020 Task 4 dataset, focusing on commonsense validation and explanation. Our methodology involves evaluating multiple LLMs, including LLaMA3-70B, Gemma2-9B, and Mixtral-8x7B, using zero-shot prompting techniques. The models are tested on two tasks: Task A (Commonsense Validation), where models determine whether a statement aligns with commonsense knowledge, and Task B (Commonsense Explanation), where models identify the reasoning behind implausible statements. Performance is assessed based on accuracy, and results are compared to fine-tuned transformer-based models. The results indicate that larger models outperform previous models and perform closely to human evaluation for Task A, with LLaMA3-70B achieving the highest accuracy of 98.40% in Task A whereas, lagging behind previous models with 93.40% in Task B. However, while models effectively identify implausible statements, they face challenges in selecting the most relevant explanation, highlighting limitations in causal and inferential reasoning.