CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm
This addresses the challenge of understanding what models learn in commonsense reasoning, which is crucial for AI researchers working on robust and generalizable AI systems, though it is incremental in benchmarking methodology.
The paper tackles the problem of evaluating models' commonsense inference capabilities by separating knowledge acquisition from inference, introducing CIKQA, a benchmark that aligns tasks with knowledge bases and converts them into a unified QA format to assess generalization across tasks.
Recently, the community has achieved substantial progress on many commonsense reasoning benchmarks. However, it is still unclear what is learned from the training process: the knowledge, inference capability, or both? We argue that due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task to cover all commonsense for learning. Thus we should separate the commonsense knowledge acquisition and inference over commonsense knowledge as two separate tasks. In this work, we focus on investigating models' commonsense inference capabilities from two perspectives: (1) Whether models can know if the knowledge they have is enough to solve the task; (2) Whether models can develop commonsense inference capabilities that generalize across commonsense tasks. We first align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not. Then, we convert different commonsense tasks into a unified question answering format to evaluate models' generalization capabilities. We name the benchmark as Commonsense Inference with Knowledge-in-the-loop Question Answering (CIKQA).