Evaluating Induced CCG Parsers on Grounded Semantic Parsing
This work addresses the need for better evaluation methods in semantic parsing for researchers, but it is incremental as it focuses on comparing existing parsers rather than introducing new techniques.
The paper tackled the problem of determining how much syntactic supervision is needed for semantic parsing by comparing four CCG parsers on a slot-filling task, resulting in the release of a new dataset called SPADES with 93K questions and making code and data publicly available.
We compare the effectiveness of four different syntactic CCG parsers for a semantic slot-filling task to explore how much syntactic supervision is required for downstream semantic analysis. This extrinsic, task-based evaluation provides a unique window to explore the strengths and weaknesses of semantics captured by unsupervised grammar induction systems. We release a new Freebase semantic parsing dataset called SPADES (Semantic PArsing of DEclarative Sentences) containing 93K cloze-style questions paired with answers. We evaluate all our models on this dataset. Our code and data are available at https://github.com/sivareddyg/graph-parser.