Independent evaluation of state-of-the-art deep networks for mammography
This highlights the need for more diverse public datasets in mammography to ensure robust model evaluation, cautioning against results without model release for independent validation.
The study evaluated five state-of-the-art deep learning models for mammography on four public datasets, finding that they performed poorly on out-of-sample data unless based on all four standard views, with performance replicating only where test data was available.
Deep neural models have shown remarkable performance in image recognition tasks, whenever large datasets of labeled images are available. The largest datasets in radiology are available for screening mammography. Recent reports, including in high impact journals, document performance of deep models at or above that of trained radiologists. What is not yet known is whether performance of these trained models is robust and replicates across datasets. Here we evaluate performance of five published state-of-the-art models on four publicly available mammography datasets. The limited size of public datasets precludes retraining the model and so we are limited to evaluate those models that have been made available with pre-trained parameters. Where test data was available, we replicated published results. However, the trained models performed poorly on out-of-sample data, except when based on all four standard views of a mammographic exam. We conclude that future progress will depend on a concerted effort to make more diverse and larger mammography datasets publicly available. Meanwhile, results that are not accompanied by a release of trained models for independent validation should be judged cautiously.