{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T04:51:09Z","timestamp":1777265469326,"version":"3.51.4"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Center for Research on Computation and Society (CRCS) at the Harvard John A. Paulson School of Engineering and Applied Sciences"},{"name":"William G. Williams Directorship at the Hospital for Sick Children"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning (ML) has the potential to transform patient care and outcomes. However, there are important differences between measuring the performance of ML models in silico and usefulness at the point of care. One lens to use to evaluate models during early development is actionability, which is currently undervalued. We propose a metric for actionability intended to be used before the evaluation of calibration and ultimately decision curve analysis and calculation of net benefit. Our metric should be viewed as part of an overarching effort to increase the number of pragmatic tools that identify a model\u2019s possible clinical impacts.<\/jats:p>","DOI":"10.1038\/s41746-023-00753-7","type":"journal-article","created":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T04:45:45Z","timestamp":1674535545000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":48,"title":["Making machine learning matter to clinicians: model actionability in medical decision-making"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6367-2865","authenticated-orcid":false,"given":"Daniel E.","family":"Ehrmann","sequence":"first","affiliation":[]},{"given":"Shalmali","family":"Joshi","sequence":"additional","affiliation":[]},{"given":"Sebastian D.","family":"Goodfellow","sequence":"additional","affiliation":[]},{"given":"Mjaye L.","family":"Mazwi","sequence":"additional","affiliation":[]},{"given":"Danny","family":"Eytan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,1,24]]},"reference":[{"key":"753_CR1","doi-asserted-by":"publisher","first-page":"1920","DOI":"10.1161\/CIRCULATIONAHA.115.001593","volume":"132","author":"RC Deo","year":"2015","unstructured":"Deo, R. C. Machine learning in medicine. Circulation 132, 1920\u20131930 (2015).","journal-title":"Circulation"},{"key":"753_CR2","doi-asserted-by":"publisher","first-page":"2281","DOI":"10.1001\/jama.2019.4914","volume":"321","author":"EJ Emanuel","year":"2019","unstructured":"Emanuel, E. J. & Wachter, R. M. Artificial intelligence in health care: will the value match the hype? JAMA 321, 2281\u20132282 (2019).","journal-title":"JAMA"},{"key":"753_CR3","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1186\/s12916-019-1426-2","volume":"17","author":"CJ Kelly","year":"2019","unstructured":"Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195\u2013195 (2019).","journal-title":"BMC Med."},{"key":"753_CR4","doi-asserted-by":"crossref","unstructured":"Shah, N. H., Milstein, A. & Bagley Ph, D. S. Making machine learning models clinically useful. JAMA 322, 1351\u20131352 (2019).","DOI":"10.1001\/jama.2019.10306"},{"key":"753_CR5","doi-asserted-by":"publisher","first-page":"i6","DOI":"10.1136\/bmj.i6","volume":"352","author":"AJ Vickers","year":"2016","unstructured":"Vickers, A. J., Van Calster, B. & Steyerberg, E. W. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 352, i6 (2016).","journal-title":"BMJ"},{"key":"753_CR6","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1016\/j.jcrc.2016.12.018","volume":"39","author":"A Razavi","year":"2017","unstructured":"Razavi, A., Newth, C. J. L., Khemani, R. G., Beltramo, F. & Ross, P. A. Cardiac output and systemic vascular resistance: clinical assessment compared with a noninvasive objective measurement in children with shock. J. Crit. Care 39, 6\u201310 (2017).","journal-title":"J. Crit. Care"},{"key":"753_CR7","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1097\/PCC.0000000000002481","volume":"21","author":"SB Walker","year":"2020","unstructured":"Walker, S. B. et al. Clinical signs to categorize shock and target vasoactive medications in warm versus cold pediatric septic shock*. Pediatr. Crit. Care Med. 21, 1051 (2020).","journal-title":"Pediatr. Crit. Care Med."},{"key":"753_CR8","doi-asserted-by":"crossref","unstructured":"Balogh, E. P., Miller, B. T., Ball, J. R. Committee on Diagnostic Error, Board on Health Care Services, et al. Overview of Diagnostic Error in Health Care. (National Academies Press, USA, 2015).","DOI":"10.17226\/21794"},{"key":"753_CR9","doi-asserted-by":"publisher","first-page":"1750","DOI":"10.1007\/s00134-001-1129-x","volume":"27","author":"M Podbregar","year":"2001","unstructured":"Podbregar, M. et al. Should we confirm our clinical diagnostic certainty by autopsies? Intensive Care Med. 27, 1750\u20131755 (2001).","journal-title":"Intensive Care Med."},{"key":"753_CR10","doi-asserted-by":"publisher","DOI":"10.1186\/s13054-017-1910-z","volume":"21","author":"J-L Vincent","year":"2017","unstructured":"Vincent, J.-L. The coming era of precision medicine for intensive care. Crit. Care 21, 314 (2017).","journal-title":"Crit. Care"},{"key":"753_CR11","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1513\/AnnalsATS.201711-867OC","volume":"15","author":"MP Kerlin","year":"2018","unstructured":"Kerlin, M. P. et al. Physician-level variation in outcomes of mechanically ventilated patients. Ann. Am. Thorac. Soc. 15, 371\u2013379 (2018).","journal-title":"Ann. Am. Thorac. Soc."},{"key":"753_CR12","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1136\/qshc.2007.023184","volume":"17","author":"JM Farnan","year":"2008","unstructured":"Farnan, J. M., Johnson, J. K., Meltzer, D. O., Humphrey, H. J. & Arora, V. M. Resident uncertainty in clinical decision making and impact on patient care: a qualitative study. Qual. Saf. Health Care 17, 122\u2013126 (2008).","journal-title":"Qual. Saf. Health Care"},{"key":"753_CR13","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s11606-017-4164-1","volume":"33","author":"V Bhise","year":"2018","unstructured":"Bhise, V. et al. Defining and measuring diagnostic uncertainty in medicine: a systematic review. J. Gen. Intern. Med. 33, 103\u2013115 (2018).","journal-title":"J. Gen. Intern. Med."},{"key":"753_CR14","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1007\/s11606-007-0159-7","volume":"22","author":"PA Bovier","year":"2007","unstructured":"Bovier, P. A. & Perneger, T. V. Stress from uncertainty from graduation to retirement-a population-based study of Swiss physicians. J. Gen. Intern. Med. 22, 632\u2013638 (2007).","journal-title":"J. Gen. Intern. Med."},{"key":"753_CR15","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1515\/dx-2014-0069","volume":"2","author":"L Zwaan","year":"2015","unstructured":"Zwaan, L. & Singh, H. The challenges in defining and measuring diagnostic error. Diagnosis 2, 97\u2013103 (2015).","journal-title":"Diagnosis"},{"key":"753_CR16","doi-asserted-by":"publisher","first-page":"1489","DOI":"10.1056\/NEJM198906013202211","volume":"320","author":"JP Kassirer","year":"1989","unstructured":"Kassirer, J. P. Our stubborn quest for diagnostic certainty. A cause of excessive testing. N. Engl. J. Med. 320, 1489\u20131491 (1989).","journal-title":"N. Engl. J. Med."},{"key":"753_CR17","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1071\/AH16088","volume":"42","author":"AM Zavala","year":"2018","unstructured":"Zavala, A. M., Day, G. E., Plummer, D. & Bamford-Wade, A. Decision-making under pressure: medical errors in uncertain and dynamic environments. Aust. Health Rev. 42, 395\u2013402 (2018).","journal-title":"Aust. Health Rev."},{"key":"753_CR18","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1001\/archinte.168.7.741","volume":"168","author":"SM Green","year":"2008","unstructured":"Green, S. M. et al. Clinical uncertainty, diagnostic accuracy, and outcomes in emergency department patients presenting with dyspnea. Arch. Intern. Med. 168, 741\u2013748 (2008).","journal-title":"Arch. Intern. Med."},{"key":"753_CR19","first-page":"887","volume":"2018","author":"X Peng","year":"2018","unstructured":"Peng, X. et al. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. AMIA Annu. Symp. Proc. 2018, 887\u2013896 (2018).","journal-title":"AMIA Annu. Symp. Proc."},{"key":"753_CR20","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1038\/s41746-021-00388-6","volume":"4","author":"A Peine","year":"2021","unstructured":"Peine, A. et al. Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care. NPJ Digit. Med. 4, 32 (2021).","journal-title":"NPJ Digit. Med."},{"key":"753_CR21","doi-asserted-by":"publisher","first-page":"e0000012","DOI":"10.1371\/journal.pdig.0000012","volume":"1","author":"T Nanayakkara","year":"2022","unstructured":"Nanayakkara, T., Clermont, G., Langmead, C. J. & Swigon, D. Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment. PLoS Digit. Health 1, e0000012 (2022).","journal-title":"PLoS Digit. Health"},{"key":"753_CR22","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","volume":"24","author":"M Komorowski","year":"2018","unstructured":"Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. & Faisal, A. A. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24, 1716\u20131720 (2018).","journal-title":"Nat. Med."},{"key":"753_CR23","first-page":"181","volume":"2020","author":"J Futoma","year":"2020","unstructured":"Futoma, J., Masood, M. A. & Doshi-Velez, F. Identifying distinct, effective treatments for acute hypotension with SODA-RL: safely optimized diverse accurate reinforcement learning. AMIA Jt. Summits Transl. Sci. Proc. 2020, 181\u2013190 (2020).","journal-title":"AMIA Jt. Summits Transl. Sci. Proc."},{"key":"753_CR24","first-page":"23","volume":"263","author":"P Krause","year":"2019","unstructured":"Krause, P. Information theory and medical decision making. Stud. Health Technol. Inform. 263, 23\u201334 (2019).","journal-title":"Stud. Health Technol. Inform."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00753-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00753-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00753-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T04:46:52Z","timestamp":1674535612000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00753-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,24]]},"references-count":24,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["753"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00753-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,24]]},"assertion":[{"value":"1 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 January 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 January 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"7"}}