Risks and NLP Design: A Case Study on Procedural Document QA
It addresses risks to users in specific NLP applications, such as injuries in cooking, but is incremental in focusing on a case study rather than broad solutions.
The paper tackles the problem of assessing and mitigating risks in NLP systems by specializing analysis to concrete applications, using cooking recipe procedural document QA as a case study, where an existing language model in zero-shot mode performs as well or better than humans on real-world questions.
As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users--and concrete strategies to mitigate them--will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.