{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T07:55:53Z","timestamp":1793346953951,"version":"4.5.0"},"reference-count":19,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T00:00:00Z","timestamp":1643760000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T00:00:00Z","timestamp":1643760000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001773","name":"University of New South Wales","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001773","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Deep neural networks are increasingly being used for computer-aided diagnosis, but erroneous diagnoses can be extremely costly for patients. We propose a learning to defer with uncertainty (LDU) algorithm which identifies patients for whom diagnostic uncertainty is high and defers them for evaluation by human experts. LDU was evaluated on the diagnosis of myocardial infarction (using discharge summaries), the diagnosis of any comorbidities (using structured data), and the diagnosis of pleural effusion and pneumothorax (using chest x-rays), and compared with \u2018learning to defer without uncertainty information\u2019 (LD) and \u2018direct triage by uncertainty\u2019 (DT) methods. LDU achieved the same F1 score as LD but deferred considerably fewer patients (e.g. 36% vs. 69% deferral rate for diagnosing pleural effusion with an F1 score of 0.96). Furthermore, even when many patients were assigned the wrong diagnosis with high confidence (e.g. for the diagnosis of any comorbidities) LDU achieved a 17% increase in F1 score, whereas DT was not applicable. Importantly, the weight of the defer loss in LDU can be easily adjusted to obtain the desired trade-off between diagnostic accuracy and deferral rate. In conclusion, LDU can readily augment any existing diagnostic network to reduce the risk of erroneous diagnoses in clinical practice.<\/jats:p>","DOI":"10.1038\/s41598-022-05725-7","type":"journal-article","created":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T06:06:03Z","timestamp":1643781963000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis"],"prefix":"10.1038","volume":"12","author":[{"given":"Jessie","family":"Liu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Blanca","family":"Gallego","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastiano","family":"Barbieri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,2,2]]},"reference":[{"key":"5725_CR1","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","volume":"577","author":"SM McKinney","year":"2020","unstructured":"McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89\u201394 (2020).","journal-title":"Nature"},{"key":"5725_CR2","unstructured":"Raghu, M. et al. The Algorithmic Automation Problem: Prediction, Triage, and Human Effort. (2019). arXiv:1903.12220"},{"key":"5725_CR3","first-page":"1321","volume":"70","author":"C Guo","year":"2017","unstructured":"Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. Int. Conf. Mach. Learn. PMLR 70, 1321\u20131330 (2017).","journal-title":"Int. Conf. Mach. Learn. PMLR"},{"key":"5725_CR4","unstructured":"Raghu, M. et al. Direct Uncertainty Prediction for Medical Second Opinions. (2018). arXiv:1807.01771"},{"key":"5725_CR5","doi-asserted-by":"crossref","unstructured":"Cortes, C., DeSalvo, G., Mohri, M. Learning with rejection. in International Conference on Algorithmic Learning Theory. 67\u201382. (Springer, 2016).","DOI":"10.1007\/978-3-319-46379-7_5"},{"key":"5725_CR6","unstructured":"Madras, D., Pitassi, T., & Zemel, R. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. 1711.06664 (2017)."},{"key":"5725_CR7","unstructured":"Mozannar, H., & Sontag, D. Consistent estimators for learning to defer to an expert. in International Conference on Machine Learning. Vol. 37. (2020)."},{"key":"5725_CR8","doi-asserted-by":"crossref","unstructured":"Zhang, X., Chen, F., Lu, C.-T., & Ramakrishnan, N. Mitigating Uncertainty in Document Classification. (2019). arXiv:1907.07590","DOI":"10.18653\/v1\/N19-1316"},{"key":"5725_CR9","unstructured":"Lakshminarayanan, B., Pritzel, A., & Blundell, C. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. (2016). arXiv:1612.01474"},{"key":"5725_CR10","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"A Johnson","year":"2016","unstructured":"Johnson, A. et al. MIMIC-III, a freely accessible critical care database. Nat. Sci. Data 3, 160035 (2016).","journal-title":"Nat. Sci. Data"},{"key":"5725_CR11","doi-asserted-by":"crossref","unstructured":"Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. (2019). arXiv:1904.03323","DOI":"10.18653\/v1\/W19-1909"},{"key":"5725_CR12","unstructured":"Mulyar, A., Schumacher, E., Rouhizadeh, M., & Dredze, M. Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models. (2019). arXiv:1910.13664"},{"key":"5725_CR13","unstructured":"Adhikari, A., Ram, A., Tang, R., & Lin, J. DocBERT: BERT for Document Classification. (2019). arXiv:1904.08398"},{"key":"5725_CR14","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","volume":"6","author":"AEW Johnson","year":"2019","unstructured":"Johnson, A. E. W. et al. MIMIC-CXR: A large publicly available database of labeled chest radiographs. Nat. Sci. Data 6, 317 (2019).","journal-title":"Nat. Sci. Data"},{"key":"5725_CR15","unstructured":"Rajpurkar, P. et al. Chexnet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. (2017). arXiv:1711.05225"},{"key":"5725_CR16","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K.Q. Densely connected convolutional networks. In IEEE CVPR. Vol. 1. 2261\u20132269. (2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"5725_CR17","unstructured":"Dissez, G., & Duboc, G. CheXpert : A large chest X-ray dataset and competition. https:\/\/github.com\/gaetandi\/cheXpert."},{"key":"5725_CR18","unstructured":"Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. Weight Uncertainty in Neural Networks. ICML. (2015)."},{"key":"5725_CR19","doi-asserted-by":"crossref","unstructured":"Bhardwaj, N., & Sharma, P. An advanced uncertainty measure using fuzzy soft sets: Application to decision-making problems. Big Data Min. Anal. (2021).","DOI":"10.26599\/BDMA.2020.9020020"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05725-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05725-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05725-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T16:43:54Z","timestamp":1669308234000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05725-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,2]]},"references-count":19,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["5725"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-05725-7","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,2]]},"assertion":[{"value":"11 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"1762"}}