Confidence Modeling for Neural Semantic Parsing
This work addresses confidence modeling for neural semantic parsing, which is incremental as it builds on existing sequence-to-sequence models to enhance reliability and interpretability.
The paper tackled the problem of uncertainty in neural semantic parsers by designing metrics to quantify three major causes and using them to estimate confidence scores, showing that their model significantly outperforms a posterior probability-based method and improves interpretation quality.
In this work we focus on confidence modeling for neural semantic parsers which are built upon sequence-to-sequence models. We outline three major causes of uncertainty, and design various metrics to quantify these factors. These metrics are then used to estimate confidence scores that indicate whether model predictions are likely to be correct. Beyond confidence estimation, we identify which parts of the input contribute to uncertain predictions allowing users to interpret their model, and verify or refine its input. Experimental results show that our confidence model significantly outperforms a widely used method that relies on posterior probability, and improves the quality of interpretation compared to simply relying on attention scores.