Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
This work addresses practical challenges in task-oriented parsing for developers and users, but it is incremental as it builds on existing confidence calibration methods.
The paper tackles the problem of balancing trade-offs in semantic parsing, such as cost versus annotator load and usability versus safety, by using calibrated confidence scores. The proposed DidYouMean system improves accuracy with few interactions and reduces incorrect program executions, though at a usability cost.
We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions. We then examine how confidence scores can help optimize the trade-off between usability and safety. We show that confidence-based thresholding can substantially reduce the number of incorrect low-confidence programs executed; however, this comes at a cost to usability. We propose the DidYouMean system which better balances usability and safety.