Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domain
This work addresses the challenge of aligning AI decisions with human ethical principles in domains like medical triage, though it is incremental in applying existing LLM techniques to a new dataset.
The paper tackles the problem of conflicting expert opinions in decision-making by introducing a dataset for medical triage labeled with decision-maker attributes, and demonstrates that large language models can be aligned to these attributes using zero-shot prompting, achieving improved performance with a weighted self-consistency method.
In difficult decision-making scenarios, it is common to have conflicting opinions among expert human decision-makers as there may not be a single right answer. Such decisions may be guided by different attributes that can be used to characterize an individual's decision. We introduce a novel dataset for medical triage decision-making, labeled with a set of decision-maker attributes (DMAs). This dataset consists of 62 scenarios, covering six different DMAs, including ethical principles such as fairness and moral desert. We present a novel software framework for human-aligned decision-making by utilizing these DMAs, paving the way for trustworthy AI with better guardrails. Specifically, we demonstrate how large language models (LLMs) can serve as ethical decision-makers, and how their decisions can be aligned to different DMAs using zero-shot prompting. Our experiments focus on different open-source models with varying sizes and training techniques, such as Falcon, Mistral, and Llama 2. Finally, we also introduce a new form of weighted self-consistency that improves the overall quantified performance. Our results provide new research directions in the use of LLMs as alignable decision-makers. The dataset and open-source software are publicly available at: https://github.com/ITM-Kitware/llm-alignable-dm.