Assessment, Criticism and Improvement of Imprecise Subjective Probabilities for a Medical Expert System
This work addresses the challenge of improving expert system reliability in medical diagnostics, though it is incremental in nature.
The study evaluated the quality of imprecise subjective probabilities provided by pediatric cardiologists for a congenital heart disease belief network, finding that while generally reliable, the probabilities tended to be too extreme, and demonstrated that combining these judgments with observed data allows for adaptive improvement.
Three paediatric cardiologists assessed nearly 1000 imprecise subjective conditional probabilities for a simple belief network representing congenital heart disease, and the quality of the assessments has been measured using prospective data on 200 babies. Quality has been assessed by a Brier scoring rule, which decomposes into terms measuring lack of discrimination and reliability. The results are displayed for each of 27 diseases and 24 questions, and generally the assessments are reliable although there was a tendency for the probabilities to be too extreme. The imprecision allows the judgements to be converted to implicit samples, and by combining with the observed data the probabilities naturally adapt with experience. This appears to be a practical procedure even for reasonably large expert systems.