{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T12:52:35Z","timestamp":1784119955994,"version":"3.55.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000050","name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL085188"],"award-info":[{"award-number":["R01HL085188"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL085188"],"award-info":[{"award-number":["R01HL085188"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute"},{"name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute"},{"name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute"},{"name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning has the potential to change the practice of medicine, particularly in areas that require pattern recognition (e.g. radiology). Although automated classification is unlikely to be perfect, few modern machine learning tools have the ability to assess their own classification confidence to recognize uncertainty that might need human review. Using automated single-channel sleep staging as a first implementation, we demonstrated that uncertainty information (as quantified using Shannon entropy) can be utilized in a \u201chuman in the loop\u201d methodology to promote targeted review of uncertain sleep stage classifications on an epoch-by-epoch basis. Across 20 sleep studies, this feedback methodology proved capable of improving scoring agreement with the gold standard over automated scoring alone (average improvement in Cohen\u2019s Kappa of 0.28), in a fraction of the scoring time compared to full manual review (60% reduction). In summary, our uncertainty-based clinician-in-the-loop framework promotes the improvement of medical classification accuracy\/confidence in a cost-effective and economically resourceful manner.<\/jats:p>","DOI":"10.1038\/s41746-021-00515-3","type":"journal-article","created":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T10:30:41Z","timestamp":1632997841000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine"],"prefix":"10.1038","volume":"4","author":[{"given":"Dae Y.","family":"Kang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pamela N.","family":"DeYoung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin","family":"Tantiongloc","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Todd P.","family":"Coleman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6515-6204","authenticated-orcid":false,"given":"Robert L.","family":"Owens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,9,30]]},"reference":[{"key":"515_CR1","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan, M. I. & Mitchell, T. M. Machine learning: trends, perspectives, and prospects. Science 349, 255\u2013260 (2015).","journal-title":"Science"},{"key":"515_CR2","doi-asserted-by":"publisher","first-page":"1920","DOI":"10.1161\/CIRCULATIONAHA.115.001593","volume":"132","author":"RC Deo","year":"2015","unstructured":"Deo, R. C. Machine learning in medicine. Circulation 132, 1920\u20131930 (2015).","journal-title":"Circulation"},{"key":"515_CR3","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1038\/s41551-018-0195-0","volume":"2","author":"Poplin","year":"2018","unstructured":"Poplin, Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158 (2018).","journal-title":"Nat. Biomed. Eng."},{"key":"515_CR4","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1001\/jamacardio.2018.0136","volume":"3","author":"GH Tison","year":"2018","unstructured":"Tison, G. H. et al. Passive detection of atrial fibrillation using a commercially available smartwatch. JAMA Cardiol. 3, 409\u2013416 (2018).","journal-title":"JAMA Cardiol."},{"key":"515_CR5","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","volume":"110","author":"E Hullermeier","year":"2021","unstructured":"Hullermeier, E. & Wageman, W. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn. 110, 457\u2013506 (2021).","journal-title":"Mach. Learn."},{"key":"515_CR6","unstructured":"Kendall, A. & Gal, Y. What uncertainties do we need in Bayesian deep learning for computer vision? Preprint at arXiv https:\/\/arxiv.org\/abs\/1703.049772015."},{"key":"515_CR7","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/j.inffus.2021.05.008","volume":"76","author":"FP Moloud Abdar","year":"2021","unstructured":"Moloud Abdar, F. P. et al. A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243\u2013297 (2021).","journal-title":"Inf. Fusion"},{"key":"515_CR8","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1002\/j.1538-7305.1948.tb00917.x","volume":"27","author":"CE Shannon","year":"1948","unstructured":"Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27, 623\u2013656 (1948).","journal-title":"Bell Syst. Tech. J."},{"key":"515_CR9","unstructured":"R\u00e9nyi, A. (1960) On Measures of Information and Entropy. Proceedings of the 4th Berkeley Symposium onMathematics, Statistics and Probability, 20 June-30 July 1960, Volume I, University of California Press, Berkeleyand Los Angeles, 547-561."},{"key":"515_CR10","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1109\/MSP.2010.936017","volume":"27","author":"W Thompson","year":"2010","unstructured":"Thompson, W., Martwick, A. & Weltman, J. Examining H1N1 through its information entropy [In the Spotlight]. IEEE Signal. Process. Mag. 27, 171\u2013174 (2010).","journal-title":"IEEE Signal. Process. Mag."},{"key":"515_CR11","first-page":"39","volume":"22","author":"A Berger","year":"1996","unstructured":"Berger, A., Della Pietra, S. A. & Della Pietra, V. J. A maximum entropy approach to natural language processing. Comput. Linguist. 22, 39\u201371 (1996).","journal-title":"Comput. Linguist."},{"key":"515_CR12","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1089\/cmb.1999.6.125","volume":"6","author":"D Loewenstern","year":"1999","unstructured":"Loewenstern, D. & Yianilos, P. N. Significantly lower entropy estimates for natural DNA sequences. J. Comput. Biol. 6, 125\u2013142 (1999).","journal-title":"J. Comput. Biol."},{"key":"515_CR13","doi-asserted-by":"publisher","first-page":"338","DOI":"10.1007\/s10278-007-9044-5","volume":"21","author":"D-Y Tsai","year":"2008","unstructured":"Tsai, D.-Y., Lee, Y. & Matsuyama, E. Information entropy measure for evaluation of image quality. J. Digital Imaging 21, 338\u2013347 (2008).","journal-title":"J. Digital Imaging"},{"key":"515_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-17876-z","volume":"7","author":"C Leibig","year":"2017","unstructured":"Leibig, C., Allken, V., Ayhan, M. S., Berens, P. & Wahl, S. Leveraging uncertainty information from deep neural networks for disease detection. Sci. Rep. 7, 17816 (2017).","journal-title":"Sci. Rep."},{"key":"515_CR15","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1038\/s41746-020-00367-3","volume":"4","author":"B Kompa","year":"2021","unstructured":"Kompa, B., Snoek, J. & Beam, A. L. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit. Med. 4, 4 (2021).","journal-title":"NPJ Digit. Med."},{"key":"515_CR16","first-page":"1","volume":"104418","author":"M Abdar","year":"2021","unstructured":"Abdar, M. Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning. Comput. Biol. Med. 104418, 1\u201317 (2021).","journal-title":"Comput. Biol. Med."},{"key":"515_CR17","unstructured":"Angelos Filos, S. F. et al. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks. Preprint at arXiv https:\/\/arxiv.org\/abs\/1912.10481 (2019)."},{"key":"515_CR18","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1038\/s41746-021-00440-5","volume":"4","author":"M Perslev","year":"2021","unstructured":"Perslev, M. et al. U-Sleep: resilient high-frequency sleep staging. NPJ Digit. Med. 4, 72 (2021).","journal-title":"NPJ Digit. Med."},{"key":"515_CR19","doi-asserted-by":"publisher","first-page":"2681","DOI":"10.1109\/TIT.2005.850223","volume":"51","author":"D Hernando","year":"2005","unstructured":"Hernando, D., Crespi, V. & Cybenko, G. Efficient computation of the hidden Markov model entropy for a given observation sequence. IEEE Trans. Inf. Theory 51, 2681\u20132685 (2005).","journal-title":"IEEE Trans. Inf. Theory"},{"key":"515_CR20","doi-asserted-by":"publisher","first-page":"2379","DOI":"10.1016\/j.csda.2006.03.015","volume":"51","author":"Y Gu\u00e9don","year":"2007","unstructured":"Gu\u00e9don, Y. Exploring the state sequence space for hidden Markov and semi-Markov chains. Comput. Stat. Data Anal. 51, 2379\u20132409 (2007).","journal-title":"Comput. Stat. Data Anal."},{"key":"515_CR21","doi-asserted-by":"publisher","first-page":"549","DOI":"10.1007\/s11222-014-9494-9","volume":"26","author":"J-B Durand","year":"2014","unstructured":"Durand, J.-B. & Gu\u00e9don, Y. Localizing the latent structure canonical uncertainty: entropy profiles for hidden Markov models. Stat. Comput. 26, 549\u2013567 (2014).","journal-title":"Stat. Comput."},{"key":"515_CR22","first-page":"1050","volume":"48","author":"Y Gal","year":"2016","unstructured":"Gal, Y. & Ghahramani, Z. Proceedings of The 33rd International Conference onMachine Learning. PMLR 48, 1050\u20131059 (2016).","journal-title":"PMLR"},{"key":"515_CR23","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1214\/aoms\/1177704567","volume":"33","author":"MH DeGroot","year":"1962","unstructured":"DeGroot, M. H. Uncertainty, information, and sequential experiments. Ann. Math. Stat. 33, 404\u2013419 (1962).","journal-title":"Ann. Math. Stat."},{"key":"515_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/sleep\/zsx139","volume":"40","author":"H Sun","year":"2017","unstructured":"Sun, H. et al. Large-scale automated sleep staging. Sleep 40, 1\u201312 (2017).","journal-title":"Sleep"},{"key":"515_CR25","first-page":"1206","volume":"12","author":"M Younes","year":"2015","unstructured":"Younes, M., Thompson, W., Leslie, C., Egan, T. & Giannouli, E. Utility of technologist editing of polysomnography scoring performed by a validated automatic system. Ann. Am. Thorac. Soc. 12, 1206\u20131218 (2015).","journal-title":"Ann. Am. Thorac. Soc."},{"key":"515_CR26","doi-asserted-by":"publisher","first-page":"245","DOI":"10.5664\/jcsm.6456","volume":"13","author":"M Younes","year":"2017","unstructured":"Younes, M., Soiferman, M., Thompson, W. & Giannouli, E. Performance of a new portable wireless sleep monitor. J. Clin. Sleep Med. 13, 245\u2013258 (2017).","journal-title":"J. Clin. Sleep Med."},{"key":"515_CR27","doi-asserted-by":"publisher","first-page":"735","DOI":"10.5664\/jcsm.5808","volume":"12","author":"M Younes","year":"2016","unstructured":"Younes, M., Younes, M. & Giannouli, E. Accuracy of automatic polysomnography scoring using frontal electrodes. J. Clin. Sleep Med. 12, 735\u2013746 (2016).","journal-title":"J. Clin. Sleep Med."},{"key":"515_CR28","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1109\/TBME.2017.2702123","volume":"65","author":"DY Kang","year":"2017","unstructured":"Kang, D. Y., DeYoung, P. N., Malhotra, A., Owens, R. L. & Coleman, T. P. A state space and density estimation framework for sleep staging in obstructive sleep apnea. IEEE Trans. Biomed. Eng. 65, 1201\u20131212 (2017).","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"515_CR29","doi-asserted-by":"publisher","first-page":"665","DOI":"10.5664\/jcsm.6576","volume":"13","author":"RB Berry","year":"2017","unstructured":"Berry, R. B. et al. AASM scoring manual updates for 2017 (Version 2.4). J. Clin. Sleep Med. 13, 665\u2013666 (2017).","journal-title":"J. Clin. Sleep Med."},{"key":"515_CR30","doi-asserted-by":"publisher","first-page":"573","DOI":"10.5665\/sleep.2548","volume":"36","author":"A Malhotra","year":"2013","unstructured":"Malhotra, A. et al. Performance of an automated polysomnography scoring system versus computer-assisted manual scoring. Sleep 36, 573\u2013582 (2013).","journal-title":"Sleep"},{"key":"515_CR31","doi-asserted-by":"publisher","first-page":"591","DOI":"10.5665\/sleep.2552","volume":"36","author":"UJ Magalang","year":"2013","unstructured":"Magalang, U. J. et al. Agreement in the scoring of respiratory events and sleep among international sleep centers. Sleep 36, 591\u2013596 (2013).","journal-title":"Sleep"},{"key":"515_CR32","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis, J. R., Richard Landis, J. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 159 (1977).","journal-title":"Biometrics"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00515-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00515-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00515-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T19:00:17Z","timestamp":1670094017000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00515-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,30]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["515"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00515-3","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,30]]},"assertion":[{"value":"23 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"142"}}