Characterizing the Value of Information in Medical Notes
This addresses data quality issues in healthcare for clinicians and researchers, but is incremental as it builds on existing probing and selection methods.
The study tackled the problem of assessing the value of information in medical notes for predictive tasks, showing that notes only add predictive power for readmission prediction over structured data, and that selecting valuable parts (6.8% of tokens) improves performance despite distribution shifts.
Machine learning models depend on the quality of input data. As electronic health records are widely adopted, the amount of data in health care is growing, along with complaints about the quality of medical notes. We use two prediction tasks, readmission prediction and in-hospital mortality prediction, to characterize the value of information in medical notes. We show that as a whole, medical notes only provide additional predictive power over structured information in readmission prediction. We further propose a probing framework to select parts of notes that enable more accurate predictions than using all notes, despite that the selected information leads to a distribution shift from the training data ("all notes"). Finally, we demonstrate that models trained on the selected valuable information achieve even better predictive performance, with only 6.8% of all the tokens for readmission prediction.