{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T20:39:46Z","timestamp":1772138386584,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2021,9,2]],"date-time":"2021-09-02T00:00:00Z","timestamp":1630540800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Fonds de Recherche du Quebec\u2014Sant\u00e9 and the Minist\u00e8re de la Sant\u00e9 et des Services Sociaux","award":["MR1-291226"],"award-info":[{"award-number":["MR1-291226"]}]},{"name":"Fonds de Recherche du Quebec\u2014Sant\u00e9 and the Minist\u00e8re de la Sant\u00e9 et des Services Sociaux","award":["CC-253453"],"award-info":[{"award-number":["CC-253453"]}]},{"name":"Fonds de Recherche du Quebec\u2014Sant\u00e9"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,10,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models\u20143 random forest models and the Modified Hospital One-year Mortality Risk model\u2014to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocab140","type":"journal-article","created":{"date-parts":[[2021,6,21]],"date-time":"2021-06-21T15:14:46Z","timestamp":1624288486000},"page":"2366-2378","source":"Crossref","is-referenced-by-count":12,"title":["Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2961-233X","authenticated-orcid":false,"given":"Ryeyan","family":"Taseen","sequence":"first","affiliation":[{"name":"Respiratory Division, Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Quebec, Canada"},{"name":"Centre Interdisciplinaire de Recherche en Informatique de la Sant\u00e9, University of Sherbrooke, Sherbrooke, Quebec, Canada"},{"name":"Groupe de Recherche Interdisciplinaire en Informatique de la Sant\u00e9, University of Sherbrooke, Sherbrooke, Quebec, Canada"}]},{"given":"Jean-Fran\u00e7ois","family":"Ethier","sequence":"additional","affiliation":[{"name":"Centre Interdisciplinaire de Recherche en Informatique de la Sant\u00e9, University of Sherbrooke, Sherbrooke, Quebec, Canada"},{"name":"Groupe de Recherche Interdisciplinaire en Informatique de la Sant\u00e9, University of Sherbrooke, Sherbrooke, Quebec, Canada"},{"name":"General Internal Medicine Division, Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Quebec, Canada"}]}],"member":"286","published-online":{"date-parts":[[2021,9,2]]},"reference":[{"key":"2021101218330280500_ocab140-B1","doi-asserted-by":"crossref","first-page":"c1345","DOI":"10.1136\/bmj.c1345","article-title":"The impact of advance care planning on end of life care in elderly patients: randomised controlled trial","volume":"340","author":"Detering","year":"2010","journal-title":"BMJ"},{"issue":"12","key":"2021101218330280500_ocab140-B2","doi-asserted-by":"crossref","first-page":"1994","DOI":"10.1001\/jamainternmed.2014.5271","article-title":"Communication about serious illness care goals: a review and synthesis of best practices","volume":"174","author":"Bernacki","year":"2014","journal-title":"JAMA Intern Med"},{"issue":"3","key":"2021101218330280500_ocab140-B3","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1177\/1049909117715217","article-title":"Utilizing the electronic health record to improve advance care planning: a systematic review","volume":"35","author":"Huber","year":"2018","journal-title":"Am J Hosp Palliat Care"},{"key":"2021101218330280500_ocab140-B4","doi-asserted-by":"crossref","first-page":"h2361","DOI":"10.1136\/bmj.h2361","article-title":"The role of intervening hospital admissions on trajectories of disability in the last year of life: prospective cohort study of older people","volume":"350","author":"Gill","year":"2015","journal-title":"BMJ"},{"issue":"9","key":"2021101218330280500_ocab140-B5","doi-asserted-by":"crossref","first-page":"778","DOI":"10.1001\/jamainternmed.2013.180","article-title":"Failure to engage hospitalized elderly patients and their families in advance care planning","volume":"173","author":"Heyland","year":"2013","journal-title":"JAMA Intern Med"},{"issue":"2","key":"2021101218330280500_ocab140-B6","doi-asserted-by":"crossref","first-page":"e0116629","DOI":"10.1371\/journal.pone.0116629","article-title":"Barriers to advance care planning at the end of life: an explanatory systematic review of implementation studies","volume":"10","author":"Lund","year":"2015","journal-title":"PloS One"},{"issue":"10225","key":"2021101218330280500_ocab140-B7","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1016\/S0140-6736(20)30318-4","article-title":"Power and perils of prediction in palliative care","volume":"395","author":"Porter","year":"2020","journal-title":"Lancet"},{"key":"2021101218330280500_ocab140-B8","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-16399-0","volume-title":"Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating","author":"Steyerberg","year":"2019"},{"key":"2021101218330280500_ocab140-B9","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1186\/s41512-018-0033-6","article-title":"Evaluating the impact of prediction models: lessons learned, challenges, and recommendations","volume":"2","author":"Kappen","year":"2018","journal-title":"Diagn Progn Res"},{"issue":"3","key":"2021101218330280500_ocab140-B10","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1097\/EDE.0000000000001173","article-title":"Aim for clinical utility, not just predictive accuracy","volume":"31","author":"Sachs","year":"2020","journal-title":"Epidemiology"},{"issue":"3","key":"2021101218330280500_ocab140-B11","doi-asserted-by":"crossref","first-page":"826","DOI":"10.4338\/ACI-2017-03-CR-0046","article-title":"Barriers to achieving economies of scale in analysis of EHR data. A cautionary tale","volume":"8","author":"Sendak","year":"2017","journal-title":"Appl Clin Inform"},{"key":"2021101218330280500_ocab140-B12","doi-asserted-by":"crossref","first-page":"2366","DOI":"10.1038\/s41746-020-0221-y","article-title":"An overview of clinical decision support systems: benefits, risks, and strategies for success","volume":"3","author":"Sutton","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"9","key":"2021101218330280500_ocab140-B13","doi-asserted-by":"crossref","first-page":"e489","DOI":"10.1016\/S2589-7500(20)30186-2","article-title":"The myth of generalisability in clinical research and machine learning in health care","volume":"2","author":"Futoma","year":"2020","journal-title":"Lancet Digit Health"},{"key":"2021101218330280500_ocab140-B14","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139506779","volume-title":"Decision Making in Health and Medicine: Integrating Evidence and Values","author":"Hunink","year":"2014","edition":"2nd ed"},{"key":"2021101218330280500_ocab140-B15","doi-asserted-by":"crossref","first-page":"i6","DOI":"10.1136\/bmj.i6","article-title":"Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests","volume":"352","author":"Vickers","year":"2016","journal-title":"BMJ"},{"issue":"12","key":"2021101218330280500_ocab140-B16","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1093\/jamia\/ocz088","article-title":"The number needed to benefit: estimating the value of predictive analytics in healthcare","volume":"26","author":"Liu","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2021101218330280500_ocab140-B17","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1053\/j.seminoncol.2009.12.004","article-title":"Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework","volume":"37","author":"Vickers","year":"2010","journal-title":"Semin Oncol"},{"issue":"14","key":"2021101218330280500_ocab140-B18","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2019.10306","article-title":"Making machine learning models clinically useful","volume":"322","author":"Shah","year":"2019","journal-title":"JAMA"},{"issue":"3","key":"2021101218330280500_ocab140-B19","doi-asserted-by":"crossref","first-page":"e168","DOI":"10.1200\/JOP.2017.023200","article-title":"Development of Imminent Mortality Predictor for Advanced Cancer (IMPAC), a tool to predict short-term mortality in hospitalized patients with advanced cancer","volume":"14","author":"Adelson","year":"2018","journal-title":"J Oncol Pract"},{"key":"2021101218330280500_ocab140-B20","first-page":"22","article-title":"A framework for making predictive models useful in practice","author":"Jung","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021101218330280500_ocab140-B21","first-page":"971","article-title":"mHOMR: a feasibility study of an automated system for identifying inpatients having an elevated risk of 1-year mortality","volume":"28","author":"Wegier","year":"2019","journal-title":"BMJ Qual Saf"},{"issue":"1","key":"2021101218330280500_ocab140-B22","doi-asserted-by":"crossref","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann Intern Med"},{"issue":"2","key":"2021101218330280500_ocab140-B23","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/j.jval.2013.02.002","article-title":"Consolidated Health Economic Evaluation Reporting Standards (CHEERS)\u2013explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force","volume":"16","author":"Husereau","year":"2013","journal-title":"Value Health"},{"issue":"30","key":"2021101218330280500_ocab140-B24","doi-asserted-by":"crossref","first-page":"E980","DOI":"10.1503\/cmaj.160515","article-title":"Validation of quality indicators for end-of-life communication: results of a multicentre survey","volume":"189","author":"Heyland","year":"2017","journal-title":"CMAJ"},{"issue":"9","key":"2021101218330280500_ocab140-B25","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1001\/jamainternmed.2013.203","article-title":"Disregard of patients\u2019 preferences is a medical error: comment on \u201cFailure to engage hospitalized elderly patients and their families in advance care planning","volume":"173","author":"Allison","year":"2013","journal-title":"JAMA Intern Med"},{"key":"2021101218330280500_ocab140-B26","first-page":"161","article-title":"Deriving the expected utility of a predictive model when the utilities are uncertain","volume":"2005","author":"Cooper","year":"2005","journal-title":"AMIA Annu Symp Proc"},{"issue":"6","key":"2021101218330280500_ocab140-B27","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1177\/0272989X06295361","article-title":"Decision curve analysis: a novel method for evaluating prediction models","volume":"26","author":"Vickers","year":"2006","journal-title":"Med Decis Making"},{"issue":"3","key":"2021101218330280500_ocab140-B28","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/S0167-6296(98)00039-3","article-title":"The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies","volume":"18","author":"Claxton","year":"1999","journal-title":"J Health Econ"},{"key":"2021101218330280500_ocab140-B29","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1214\/ss\/1032280214","article-title":"Bootstrap confidence intervals","volume":"11","author":"DiCiccio","year":"1996","journal-title":"Stat Sci"},{"key":"2021101218330280500_ocab140-B30","year":"2020"},{"issue":"11","key":"2021101218330280500_ocab140-B31","doi-asserted-by":"crossref","first-page":"1130","DOI":"10.1097\/01.mlr.0000182534.19832.83","article-title":"Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data","volume":"43","author":"Quan","year":"2005","journal-title":"Med Care"},{"key":"2021101218330280500_ocab140-B32","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s41512-019-0064-7","article-title":"A simple, step-by-step guide to interpreting decision curve analysis","volume":"3","author":"Vickers","year":"2019","journal-title":"Diagn Progn Res"},{"issue":"10","key":"2021101218330280500_ocab140-B33","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1503\/cmaj.150209","article-title":"External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission","volume":"187","author":"van","year":"2015","journal-title":"CMAJ"},{"issue":"9","key":"2021101218330280500_ocab140-B34","doi-asserted-by":"crossref","first-page":"e1911146","DOI":"10.1001\/jamanetworkopen.2019.11146","article-title":"Early identification of people who would benefit from a palliative approach-moving from surprise to routine","volume":"2","author":"Downar","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"2","key":"2021101218330280500_ocab140-B35","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1093\/jamiaopen\/ooy028","article-title":"A systematic review of the use of the electronic health record for patient identification, communication, and clinical support in palliative care","volume":"1","author":"Bush","year":"2018","journal-title":"JAMIA Open"},{"issue":"1","key":"2021101218330280500_ocab140-B36","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1186\/s12911-020-01235-6","article-title":"Development, implementation, and prospective validation of a model to predict 60-day end-of-life in hospitalized adults upon admission at three sites","volume":"20","author":"Major","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021101218330280500_ocab140-B37","doi-asserted-by":"crossref","DOI":"10.1093\/jamia\/ocaa211","article-title":"Improving the delivery of palliative care through predictive modeling and healthcare informatics","author":"Murphree","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"2021101218330280500_ocab140-B38","doi-asserted-by":"crossref","first-page":"1841","DOI":"10.1007\/s11606-019-05169-2","article-title":"Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: a pilot quasi-experimental study","volume":"34","author":"Courtright","year":"2019","journal-title":"J Gen Intern Med"},{"issue":"10","key":"2021101218330280500_ocab140-B39","doi-asserted-by":"crossref","first-page":"e1915997","DOI":"10.1001\/jamanetworkopen.2019.15997","article-title":"Machine learning approaches to predict 6-month mortality among patients with cancer","volume":"2","author":"Parikh","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"Suppl 4","key":"2021101218330280500_ocab140-B40","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1186\/s12911-018-0677-8","article-title":"Improving palliative care with deep learning","volume":"18","author":"Avati","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"issue":"7","key":"2021101218330280500_ocab140-B41","doi-asserted-by":"crossref","first-page":"e196972","DOI":"10.1001\/jamanetworkopen.2019.6972","article-title":"Development and validation of a deep learning algorithm for mortality prediction in selecting patients with dementia for earlier palliative care interventions","volume":"2","author":"Wang","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2021101218330280500_ocab140-B42","doi-asserted-by":"crossref","first-page":"m1328","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal","volume":"369","author":"Wynants","year":"2020","journal-title":"BMJ"},{"issue":"6","key":"2021101218330280500_ocab140-B43","first-page":"e271\u201397","article-title":"A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis","volume":"1","author":"Liu","year":"2019","journal-title":"Lancet Digit Health"},{"issue":"20","key":"2021101218330280500_ocab140-B44","doi-asserted-by":"crossref","first-page":"1109","DOI":"10.1056\/NEJM198005153022003","article-title":"The threshold approach to clinical decision making","volume":"302","author":"Pauker","year":"1980","journal-title":"N Engl J Med"},{"key":"2021101218330280500_ocab140-B45","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1186\/1472-6947-8-53","article-title":"Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers","volume":"8","author":"Vickers","year":"2008","journal-title":"BMC Med Inform Decis Mak"},{"issue":"12","key":"2021101218330280500_ocab140-B46","doi-asserted-by":"crossref","first-page":"e1499","DOI":"10.1200\/JOP.19.00743","article-title":"Conducting goals-of-care discussions takes less time than imagined","volume":"16","author":"Pintova","year":"2020","journal-title":"JCO Oncol Pract"},{"issue":"10","key":"2021101218330280500_ocab140-B47","doi-asserted-by":"crossref","first-page":"950","DOI":"10.1001\/jama.2019.22523","article-title":"Association of physician orders for life-sustaining treatment with ICU admission among patients hospitalized near the end of life","volume":"323","author":"Lee","year":"2020","journal-title":"JAMA"},{"issue":"12","key":"2021101218330280500_ocab140-B48","doi-asserted-by":"crossref","first-page":"e204759","DOI":"10.1001\/jamaoncol.2020.4759","article-title":"Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial","volume":"6","author":"Manz","year":"2020","journal-title":"JAMA Oncol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/11\/2366\/40576621\/ocab140.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/11\/2366\/40576621\/ocab140.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,17]],"date-time":"2021-10-17T18:53:31Z","timestamp":1634496811000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/11\/2366\/6362565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,2]]},"references-count":48,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,9,2]]},"published-print":{"date-parts":[[2021,10,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab140","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.03.27.21254465","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,11,1]]},"published":{"date-parts":[[2021,9,2]]}}}