{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T03:16:29Z","timestamp":1773717389864,"version":"3.50.1"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2020,12,23]],"date-time":"2020-12-23T00:00:00Z","timestamp":1608681600000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Stanford Medicine Department of Medicine, an endowment from Debra and Mark Leslie, and innovations funds from Stanford Healthcare"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,6,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models\u2019 predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model\u2019s predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa318","type":"journal-article","created":{"date-parts":[[2020,11,27]],"date-time":"2020-11-27T15:11:13Z","timestamp":1606489873000},"page":"1149-1158","source":"Crossref","is-referenced-by-count":68,"title":["A framework for making predictive models useful in practice"],"prefix":"10.1093","volume":"28","author":[{"given":"Kenneth","family":"Jung","sequence":"first","affiliation":[{"name":"Stanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA"}]},{"given":"Sehj","family":"Kashyap","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA"}]},{"given":"Anand","family":"Avati","sequence":"additional","affiliation":[{"name":"Department of Computer Science, School of Engineering, Stanford University, Stanford, California, USA"}]},{"given":"Stephanie","family":"Harman","sequence":"additional","affiliation":[{"name":"Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA"}]},{"given":"Heather","family":"Shaw","sequence":"additional","affiliation":[{"name":"Stanford Healthcare, Stanford, California, USA"}]},{"given":"Ron","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA"}]},{"given":"Margaret","family":"Smith","sequence":"additional","affiliation":[{"name":"Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA"}]},{"given":"Kenny","family":"Shum","sequence":"additional","affiliation":[{"name":"Department of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA"}]},{"given":"Jacob","family":"Javitz","sequence":"additional","affiliation":[{"name":"Department of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA"}]},{"given":"Yohan","family":"Vetteth","sequence":"additional","affiliation":[{"name":"Department of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA"}]},{"given":"Tina","family":"Seto","sequence":"additional","affiliation":[{"name":"Department of Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2187-7955","authenticated-orcid":false,"given":"Steven C","family":"Bagley","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9385-7158","authenticated-orcid":false,"given":"Nigam H","family":"Shah","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics, School of Medicine, Stanford University, Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2020,12,22]]},"reference":[{"issue":"1","key":"2021061318592307900_ocaa318-B1","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"Rajkomar","year":"2018","journal-title":"NPJ Digital Med"},{"issue":"14","key":"2021061318592307900_ocaa318-B2","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"Rajkomar","year":"2019","journal-title":"N Engl J Med"},{"issue":"1","key":"2021061318592307900_ocaa318-B3","doi-asserted-by":"crossref","DOI":"10.1056\/CAT.19.1090","article-title":"Adoption of artificial intelligence and machine learning is increasing, but irrational exuberance remains","volume":"1","author":"Obermeyer","year":"2020","journal-title":"NEJM Catalyst"},{"issue":"13","key":"2021061318592307900_ocaa318-B4","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1001\/jama.2017.18391","article-title":"Big data and machine learning in health care","volume":"319","author":"Beam","year":"2018","journal-title":"JAMA"},{"issue":"1","key":"2021061318592307900_ocaa318-B5","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2021061318592307900_ocaa318-B6","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1136\/bmjinnov-2019-000359","article-title":"Bridging the implementation gap of machine learning in healthcare","volume":"6","author":"Seneviratne","year":"2020","journal-title":"BMJ Innov"},{"issue":"23","key":"2021061318592307900_ocaa318-B7","doi-asserted-by":"crossref","first-page":"2281","DOI":"10.1001\/jama.2019.4914","article-title":"Artificial intelligence in health care: Will the value match the hype?","volume":"321","author":"Emanuel","year":"2019","journal-title":"JAMA"},{"issue":"14","key":"2021061318592307900_ocaa318-B8","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2019.10306","article-title":"Making machine learning models clinically useful","volume":"322","author":"Shah","year":"2019","journal-title":"JAMA"},{"issue":"1","key":"2021061318592307900_ocaa318-B9","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41591-019-0651-8","article-title":"Estimate the hidden deployment cost of predictive models to improve patient care","volume":"26","author":"Morse","year":"2020","journal-title":"Nat Med"},{"issue":"03","key":"2021061318592307900_ocaa318-B10","doi-asserted-by":"crossref","first-page":"826","DOI":"10.4338\/ACI-2017-03-CR-0046","article-title":"Barriers to achieving economies of scale in analysis of EHR data: a cautionary tale","volume":"08","author":"Sendak","year":"2017","journal-title":"Appl Clin Inform"},{"issue":"1","key":"2021061318592307900_ocaa318-B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5334\/egems.287","article-title":"Machine learning in health care: A critical appraisal of challenges and opportunities","volume":"7","author":"Sendak","year":"2019","journal-title":"eGEMs"},{"issue":"11","key":"2021061318592307900_ocaa318-B12","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care\u2014addressing ethical challenges","volume":"378","author":"Char","year":"2018","journal-title":"N Engl J Med"},{"issue":"6396","key":"2021061318592307900_ocaa318-B13","doi-asserted-by":"crossref","first-page":"1462","DOI":"10.1126\/science.aar5045","article-title":"Predictive modeling of US health care spending in late life","volume":"360","author":"Einav","year":"2018","journal-title":"Science"},{"issue":"6464","key":"2021061318592307900_ocaa318-B14","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"2","key":"2021061318592307900_ocaa318-B15","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1089\/jpm.2007.0055","article-title":"Impact of an inpatient palliative care team: a randomized controlled trial","volume":"11","author":"Gade","year":"2008","journal-title":"J Palliat Med"},{"key":"2021061318592307900_ocaa318-B16","doi-asserted-by":"crossref","first-page":"1707","DOI":"10.1097\/CCM.0000000000004016","article-title":"Early palliative care consultation in the medical ICU: a cluster randomized crossover trial","author":"Ma","year":"2019","journal-title":"Crit Care Med"},{"issue":"2","key":"2021061318592307900_ocaa318-B17","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1177\/0269216313493466","article-title":"Evidence on the cost and cost-effectiveness of palliative care: a literature review","volume":"28","author":"Smith","year":"2014","journal-title":"Palliat Med"},{"issue":"8","key":"2021061318592307900_ocaa318-B18","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1056\/NEJMoa1000678","article-title":"Early palliative care for patients with metastatic non-small-cell lung cancer","volume":"363","author":"Temel","year":"2010","journal-title":"N Engl J Med"},{"issue":"12","key":"2021061318592307900_ocaa318-B19","doi-asserted-by":"crossref","first-page":"1702","DOI":"10.1001\/jamaoncol.2019.3105","article-title":"Association of early palliative care use with survival and place of death among patients with advanced lung cancer receiving care in the Veterans Health Administration","volume":"5","author":"Sullivan","year":"2019","journal-title":"JAMA Oncol"},{"issue":"3","key":"2021061318592307900_ocaa318-B20","first-page":"50","article-title":"The value of palliative care","volume":"67","author":"Verret","year":"2013","journal-title":"Healthc Finance Manage"},{"issue":"3","key":"2021061318592307900_ocaa318-B21","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1177\/0825859718777320","article-title":"Palliative care use among patients with solid cancer tumors: a national cancer data base study","volume":"33","author":"Osagiede","year":"2018","journal-title":"J Palliat Care"},{"issue":"2","key":"2021061318592307900_ocaa318-B22","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1177\/1049909119864025","article-title":"Palliative care consultation is underutilized in critically ill general surgery patients","volume":"37","author":"Evans","year":"2020","journal-title":"Am J Hosp Palliat Care"},{"issue":"4","key":"2021061318592307900_ocaa318-B23","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1177\/1049909118809975","article-title":"Palliative care consultation trends among hospitalized patients with advanced cancer in the United States, 2005 to 2014","volume":"36","author":"Rubens","year":"2019","journal-title":"Am J Hosp Palliat Care"},{"issue":"7","key":"2021061318592307900_ocaa318-B24","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1016\/j.tcm.2018.02.008","article-title":"Palliative care in heart failure","volume":"28","author":"Wiskar","year":"2018","journal-title":"Trends Cardiovasc Med"},{"key":"2021061318592307900_ocaa318-B25","author":"Avati"},{"key":"2021061318592307900_ocaa318-B26","author":"Avati","year":"2019"},{"issue":"10","key":"2021061318592307900_ocaa318-B27","doi-asserted-by":"crossref","first-page":"e1915997","DOI":"10.1001\/jamanetworkopen.2019.15997","article-title":"Machine learning approaches to predict 6-month mortality among patients with cancer","volume":"2","author":"Parikh","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"9","key":"2021061318592307900_ocaa318-B28","doi-asserted-by":"crossref","first-page":"1841","DOI":"10.1007\/s11606-019-05169-2","article-title":"Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: a pilot quasi-experimental study","volume":"34","author":"Courtright","year":"2019","journal-title":"J Gen Intern MED"},{"issue":"12","key":"2021061318592307900_ocaa318-B29","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1093\/jamia\/ocz088","article-title":"The number needed to benefit: estimating the value of predictive analytics in healthcare","volume":"26","author":"Liu","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2021061318592307900_ocaa318-B30","doi-asserted-by":"crossref","first-page":"357","DOI":"10.2165\/00019053-199915040-00003","article-title":"Recent advances in the methods of cost-benefit analysis in healthcare. Matching the art to the science","volume":"15","author":"McIntosh","year":"1999","journal-title":"Pharmacoeconomics"},{"key":"2021061318592307900_ocaa318-B31","first-page":"1076","article-title":"Interactive cost-benefit analysis: providing real-world financial context to predictive analytics","volume":"2018","author":"Weiner","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"issue":"7","key":"2021061318592307900_ocaa318-B32","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1377\/hlthaff.2014.0041","article-title":"Big data in health care: using analytics to identify and manage high-risk and high-cost patients","volume":"33","author":"Bates","year":"2014","journal-title":"Health Aff"},{"key":"2021061318592307900_ocaa318-B33","author":"E","year":"2020"},{"issue":"22","key":"2021061318592307900_ocaa318-B34","doi-asserted-by":"crossref","first-page":"2038","DOI":"10.1001\/archinternmed.2010.460","article-title":"Hospital-based palliative medicine consultation: a randomized controlled trial","volume":"170","author":"Pantilat","year":"2010","journal-title":"Arch Intern Med"},{"issue":"6","key":"2021061318592307900_ocaa318-B35","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1177\/0272989X06295361","article-title":"Decision curve analysis: a novel method for evaluating prediction models","volume":"26","author":"Vickers","year":"2006","journal-title":"Med Decis Making"},{"key":"2021061318592307900_ocaa318-B36","first-page":"3149","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17)","author":"Ke","year":"2017"},{"key":"2021061318592307900_ocaa318-B37","first-page":"391","article-title":"STRIDE\u2014An integrated standards-based translational research informatics platform","volume":"2009","author":"Lowe","year":"2009","journal-title":"AMIA Annu Symp Proc"},{"key":"2021061318592307900_ocaa318-B38","year":"2020"},{"key":"2021061318592307900_ocaa318-B39","year":"2020"},{"issue":"11","key":"2021061318592307900_ocaa318-B40","doi-asserted-by":"crossref","first-page":"3010","DOI":"10.1016\/j.csda.2011.05.006","article-title":"Model-based SIR for dimension reduction","volume":"55","author":"Scrucca","year":"2011","journal-title":"Comput Stat Data Anal"},{"issue":"1","key":"2021061318592307900_ocaa318-B41","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1038\/s41591-019-0727-5","article-title":"Axes of a revolution: challenges and promises of big data in healthcare","volume":"26","author":"Shilo","year":"2020","journal-title":"Nat Med"},{"issue":"24","key":"2021061318592307900_ocaa318-B42","doi-asserted-by":"crossref","first-page":"2405","DOI":"10.1001\/jama.2019.5284","article-title":"The proliferation of reports on clinical scoring systems: issues about uptake and clinical utility","volume":"321","author":"Challener","year":"2019","journal-title":"JAMA"},{"issue":"5","key":"2021061318592307900_ocaa318-B43","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1002\/jhm.1929","article-title":"Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record","volume":"7","author":"Escobar","year":"2012","journal-title":"J Hosp Med"},{"key":"2021061318592307900_ocaa318-B44","doi-asserted-by":"crossref","first-page":"S5","DOI":"10.1002\/jhm.2653","article-title":"Early detection, prevention, and mitigation of critical illness outside intensive care settings","volume":"11 (Suppl 1","author":"Escobar","year":"2016","journal-title":"J Hosp Med"},{"key":"2021061318592307900_ocaa318-B45","doi-asserted-by":"crossref","first-page":"S18","DOI":"10.1002\/jhm.2652","article-title":"Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals","volume":"11 (Suppl 1","author":"Escobar","year":"2016","journal-title":"J Hosp Med"},{"key":"2021061318592307900_ocaa318-B46","doi-asserted-by":"crossref","first-page":"S25","DOI":"10.1002\/jhm.2661","article-title":"Incorporating an early detection system into routine clinical practice in two community hospitals","volume":"11 (Suppl 1","author":"Dummett","year":"2016","journal-title":"J Hosp Med"},{"issue":"3","key":"2021061318592307900_ocaa318-B47","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1016\/j.jval.2017.01.013","article-title":"Constrained optimization methods in health services research-an introduction: Report 1 of the ISPOR optimization methods emerging good practices task force","volume":"20","author":"Crown","year":"2017","journal-title":"Value Health"},{"issue":"9","key":"2021061318592307900_ocaa318-B48","doi-asserted-by":"crossref","first-page":"1019","DOI":"10.1016\/j.jval.2018.05.003","article-title":"Application of constrained optimization methods in health services research: report 2 of the ISPOR optimization methods emerging good practices task force","volume":"21","author":"Crown","year":"2018","journal-title":"Value Health"},{"key":"2021061318592307900_ocaa318-B49","author":"Beede"},{"issue":"1","key":"2021061318592307900_ocaa318-B50","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-020-00318-y","article-title":"Developing a delivery science for","volume":"3","author":"Li","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"10","key":"2021061318592307900_ocaa318-B51","doi-asserted-by":"crossref","first-page":"e1914051","DOI":"10.1001\/jamanetworkopen.2019.14051","article-title":"Trends and focus of machine learning applications for health research","volume":"2","author":"Beaulieu-Jones","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2021061318592307900_ocaa318-B52","author":"Dalca"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1149\/38615384\/ocaa318.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1149\/38615384\/ocaa318.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,13]],"date-time":"2021-06-13T15:00:18Z","timestamp":1623596418000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/6\/1149\/6045012"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,22]]},"references-count":52,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,12,22]]},"published-print":{"date-parts":[[2021,6,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa318","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.07.10.20149419","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,6,1]]},"published":{"date-parts":[[2020,12,22]]}}}