Design Considerations For Hypothesis Rejection Modules In Spoken Language Understanding Systems
This work addresses a specific challenge in SLU systems for improving reliability by rejecting erroneous outputs, but it is incremental as it builds on existing rejection concepts with minor variations.
The paper tackles the problem of discarding incorrect hypotheses in Spoken Language Understanding (SLU) systems by presenting two rejection module designs, R1 and R2, which achieve similar results (e.g., 2.5% FRR at 4.5-4.6% FAR), with improvements seen when incorporating ASR features (e.g., 1.9% FRR at 3.8% FAR).
Spoken Language Understanding (SLU) systems typically consist of a set of machine learning models that operate in conjunction to produce an SLU hypothesis. The generated hypothesis is then sent to downstream components for further action. However, it is desirable to discard an incorrect hypothesis before sending it downstream. In this work, we present two designs for SLU hypothesis rejection modules: (i) scheme R1 that performs rejection on domain specific SLU hypothesis and, (ii) scheme R2 that performs rejection on hypothesis generated from the overall SLU system. Hypothesis rejection modules in both schemes reject/accept a hypothesis based on features drawn from the utterance directed to the SLU system, the associated SLU hypothesis and SLU confidence score. Our experiments suggest that both the schemes yield similar results (scheme R1: 2.5% FRR @ 4.5% FAR, scheme R2: 2.5% FRR @ 4.6% FAR), with the best performing systems using all the available features. We argue that while either of the rejection schemes can be chosen over the other, they carry some inherent differences which need to be considered while making this choice. Additionally, we incorporate ASR features in the rejection module (obtaining an 1.9% FRR @ 3.8% FAR) and analyze the improvements.