{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T22:45:14Z","timestamp":1776379514926,"version":"3.51.2"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"This study was sponsored by Flatiron Health Inc., which is an independent subsidiary of the Roche Group."},{"DOI":"10.13039\/100000056","name":"U.S. Department of Health & Human Services | NIH | National Institute of Nursing Research","doi-asserted-by":"publisher","award":["T32NR013456"],"award-info":[{"award-number":["T32NR013456"]}],"id":[{"id":"10.13039\/100000056","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We present a general framework for developing a machine learning (ML) tool that supports clinician assessment of patient risk using electronic health record-derived real-world data and apply the framework to a quality improvement use case in an oncology setting to identify patients at risk for a near-term (60 day) emergency department (ED) visit who could potentially be eligible for a home-based acute care program. Framework steps include defining clinical quality improvement goals, model development and validation, bias assessment, retrospective and prospective validation, and deployment in clinical workflow. In the retrospective analysis for the use case, 8% of patient encounters were associated with a high risk (pre-defined as predicted probability \u226520%) for a near-term ED visit by the patient. Positive predictive value (PPV) and negative predictive value (NPV) for future ED events was 26% and 91%, respectively. Odds ratio (OR) of ED visit (high- vs. low-risk) was 3.5 (95% CI: 3.4\u20133.5). The model appeared to be calibrated across racial, gender, and ethnic groups. In the prospective analysis, 10% of patients were classified as high risk, 76% of whom were confirmed by clinicians as eligible for home-based acute care. PPV and NPV for future ED events was 22% and 95%, respectively. OR of ED visit (high- vs. low-risk) was 5.4 (95% CI: 2.6\u201311.0). The proposed framework for an ML-based tool that supports clinician assessment of patient risk is a stepwise development approach; we successfully applied the framework to an ED visit risk prediction use case.<\/jats:p>","DOI":"10.1038\/s41746-022-00660-3","type":"journal-article","created":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T14:04:33Z","timestamp":1660658673000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["A machine learning framework supporting prospective clinical decisions applied to risk prediction in oncology"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1096-8974","authenticated-orcid":false,"given":"Lorinda","family":"Coombs","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abigail","family":"Orlando","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5184-2935","authenticated-orcid":false,"given":"Xiaoliang","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pooja","family":"Shaw","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander S.","family":"Rich","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shreyas","family":"Lakhtakia","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karen","family":"Titchener","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4251-2912","authenticated-orcid":false,"given":"Blythe","family":"Adamson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebecca A.","family":"Miksad","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kathi","family":"Mooney","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"660_CR1","doi-asserted-by":"publisher","first-page":"1347","DOI":"10.1056\/NEJMra1814259","volume":"380","author":"A Rajkomar","year":"2019","unstructured":"Rajkomar, A., Dean, J. & Kohane, I. Machine learning in medicine. N. Engl. J. Med. 380, 1347\u20131358 (2019).","journal-title":"N. Engl. J. Med."},{"key":"660_CR2","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1093\/jamia\/ocw042","volume":"24","author":"BA Goldstein","year":"2017","unstructured":"Goldstein, B. A., Navar, A. M., Pencina, M. J. & Ioannidis, J. P. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J. Am. Med. Inform. Assoc. 24, 198\u2013208 (2017).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"660_CR3","doi-asserted-by":"publisher","first-page":"2859","DOI":"10.1118\/1.3582947","volume":"38","author":"A Pella","year":"2011","unstructured":"Pella, A. et al. Use of machine learning methods for prediction of acute toxicity in organs at risk following prostate radiotherapy. Med. Phys. 38, 2859\u20132867 (2011).","journal-title":"Med. Phys."},{"key":"660_CR4","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1001\/jamacardio.2016.3956","volume":"2","author":"JD Frizzell","year":"2017","unstructured":"Frizzell, J. D. et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: Comparison of machine learning and other statistical approaches. JAMA Cardiol. 2, 204\u2013209 (2017).","journal-title":"JAMA Cardiol."},{"key":"660_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1200\/CCI.18.00037","volume":"2","author":"JC Hong","year":"2018","unstructured":"Hong, J. C., Niedzwiecki, D., Palta, M. & Tenenbaum, J. D. Predicting emergency visits and hospital admissions during radiation and chemoradiation: an internally validated pretreatment machine learning algorithm. JCO Clin. Cancer Inform. 2, 1\u201311 (2018).","journal-title":"JCO Clin. Cancer Inform."},{"key":"660_CR6","doi-asserted-by":"publisher","first-page":"e7","DOI":"10.37765\/ajmc.2020.42147","volume":"26","author":"K Morawski","year":"2020","unstructured":"Morawski, K., Dvorkis, Y. & Monsen, C. B. Predicting hospitalizations from electronic health record data. Am. J. Manag. Care. 26, e7\u2013e13 (2020).","journal-title":"Am. J. Manag. Care."},{"key":"660_CR7","doi-asserted-by":"publisher","first-page":"921","DOI":"10.1007\/s11606-018-4316-y","volume":"33","author":"N Sahni","year":"2018","unstructured":"Sahni, N., Simon, G. & Arora, R. Development and validation of machine learning models for prediction of 1-year mortality utilizing electronic medical record data available at the end of hospitalization in multicondition patients: a proof-of-concept study. J. Gen. Intern. Med. 33, 921\u2013928 (2018).","journal-title":"J. Gen. Intern. Med."},{"key":"660_CR8","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1016\/j.jpedsurg.2020.10.021","volume":"56","author":"N Shahi","year":"2021","unstructured":"Shahi, N. et al. Decision-making in pediatric blunt solid organ injury: a deep learning approach to predict massive transfusion, need for operative management, and mortality risk. J. Pediatr. Surg. 56, 379\u2013384 (2021).","journal-title":"J. Pediatr. Surg."},{"key":"660_CR9","doi-asserted-by":"publisher","first-page":"e158","DOI":"10.1016\/S2589-7500(20)30314-9","volume":"3","author":"C Lee","year":"2021","unstructured":"Lee, C. et al. Application of a novel machine learning framework for predicting non-metastatic prostate cancer-specific mortality in men using the surveillance, epidemiology, and end results (SEER) database. Lancet Digit. Health 3, e158\u2013e165 (2021).","journal-title":"Lancet Digit. Health"},{"key":"660_CR10","doi-asserted-by":"publisher","unstructured":"Ben-Israel, D. et al. The impact of machine learning on patient care: a systematic review. Artif. Intell. Med. https:\/\/doi.org\/10.1016\/j.artmed.2019.101785 (2020).","DOI":"10.1016\/j.artmed.2019.101785"},{"key":"660_CR11","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1038\/s42256-020-00239-1","volume":"2","author":"BA Mateen","year":"2020","unstructured":"Mateen, B. A., Liley, J., Denniston, A. K., Holmes, C. C. & Vollmer, S. J. Improving the quality of machine learning in health applications and clinical research. Nat. Mach. Intell. 2, 554\u2013556 (2020).","journal-title":"Nat. Mach. Intell."},{"key":"660_CR12","unstructured":"WhiteHouse.Gov. Notice on the continuation of the national emergency concerning the coronavirus disease 2019 (COVID-19) pandemic. https:\/\/www.whitehouse.gov\/briefing-room\/presidential-actions\/2021\/02\/24\/notice-on-the-continuation-of-the-national-emergency-concerning-the-coronavirus-disease-2019-covid-19-pandemic\/ (2021)."},{"key":"660_CR13","doi-asserted-by":"publisher","first-page":"410","DOI":"10.1038\/s41563-019-0345-0","volume":"18","author":"PC Chen","year":"2019","unstructured":"Chen, P. C., Liu, Y. & Peng, L. How to develop machine learning models for healthcare. Nat. Mater. 18, 410\u2013414 (2019).","journal-title":"Nat. Mater."},{"key":"660_CR14","doi-asserted-by":"publisher","unstructured":"Vollmer, S. et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ. https:\/\/doi.org\/10.1136\/bmj.l6927 (2020).","DOI":"10.1136\/bmj.l6927"},{"key":"660_CR15","doi-asserted-by":"publisher","first-page":"1448","DOI":"10.1093\/jamia\/ocz127","volume":"26","author":"SE Davis","year":"2019","unstructured":"Davis, S. E. et al. A nonparametric updating method to correct clinical prediction model drift. J. Am. Med. Inform. Assoc. 26, 1448\u20131457 (2019).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"660_CR16","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1177\/0962280215626466","volume":"27","author":"TL Su","year":"2018","unstructured":"Su, T. L., Jaki, T., Hickey, G. L., Buchan, I. & Sperrin, M. A review of statistical updating methods for clinical prediction models. Stat. Methods Med. Res. 27, 185\u2013197 (2018).","journal-title":"Stat. Methods Med. Res."},{"key":"660_CR17","doi-asserted-by":"publisher","first-page":"2586","DOI":"10.1200\/JCO.20.03609","volume":"39","author":"K Mooney","year":"2021","unstructured":"Mooney, K. et al. Evaluation of oncology hospital at home: Unplanned health care utilization and costs in the huntsman at home real-world trial. JCO 39, 2586\u20132593 (2021).","journal-title":"JCO"},{"key":"660_CR18","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1177\/0272989X12454579","volume":"32","author":"DM Eddy","year":"2012","unstructured":"Eddy, D. M. et al. Model transparency and validation: A report of the ISPOR-SMDM modeling good research practices task force-7. Med. Decis. Mak. 32, 733\u2013743 (2012).","journal-title":"Med. Decis. Mak."},{"key":"660_CR19","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.1038\/s41591-020-1041-y","volume":"26","author":"B Norgeot","year":"2020","unstructured":"Norgeot, B. et al. Minimum information about clinical artificial intelligence modeling: The MI-CLAIM checklist. Nat. Med. 26, 1320\u20131324 (2020).","journal-title":"Nat. Med."},{"key":"660_CR20","doi-asserted-by":"publisher","first-page":"7000","DOI":"10.1200\/JCO.2020.38.15_suppl.7000","volume":"38","author":"K Mooney","year":"2020","unstructured":"Mooney, K. et al. The oncology hospital at home: Health care utilization outcomes from the huntsman at home trial. JCO 38, 7000 (2020).","journal-title":"JCO"},{"key":"660_CR21","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1257\/aer.p20171084","volume":"107","author":"S Mullainathan","year":"2017","unstructured":"Mullainathan, S. & Obermeyer, Z. Does machine learning automate moral hazard and error? Am. Econ. Rev. 107, 476\u2013480 (2017).","journal-title":"Am. Econ. Rev."},{"key":"660_CR22","unstructured":"Birnbaum, B. et al. Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research. Preprint at https:\/\/arxiv.org\/abs\/2001.09765 (2020)."},{"key":"660_CR23","doi-asserted-by":"crossref","unstructured":"Ma, X., Long, L., Moon, S., Adamson, B. J. S. & Baxi, S. S. Comparison of population characteristics in real-world clinical oncology databases in the US: Flatiron health, SEER, and NPCR. Preprint at https:\/\/www.medrxiv.org\/content\/10.1101\/2020.03.16.20037143v2 (2020).","DOI":"10.1101\/2020.03.16.20037143"},{"key":"660_CR24","unstructured":"Corbett-Davis, S. & Goel, S. The measure and mismeasure of fairness: a critical review of fair machine learning. Preprint at https:\/\/arxiv.org\/abs\/1808.00023 (2018)."},{"key":"660_CR25","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","volume":"366","author":"Z Obermeyer","year":"2019","unstructured":"Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447 (2019).","journal-title":"Science"},{"key":"660_CR26","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L. & Topic Group \u2018Evaluating diagnostic tests and prediction models\u2019 of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med. 17, 230 (2019).","journal-title":"BMC Med."},{"key":"660_CR27","first-page":"890","volume":"31","author":"Y Bathaee","year":"2018","unstructured":"Bathaee, Y. The artificial intelligence black box and the failure of intent and causation. Harv. J. Law Technol. 31, 890\u2013938 (2018).","journal-title":"Harv. J. Law Technol."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00660-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00660-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00660-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T08:14:19Z","timestamp":1669364059000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00660-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,16]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["660"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00660-3","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,16]]},"assertion":[{"value":"20 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing non-financial interests but the following competing financial interests: A.O., X.W., P.S., A.S.R., S.L., B.A., and R.A.M. report employment at Flatiron Health, Inc., which is an independent subsidiary of the Roche Group. A.O., X.W., P.S., A.S.R., S.L., B.A., and R.A.M. also report stock ownership in Roche. A.O., P.S., R.A.M. also report equity ownership in Flatiron Health, Inc. L.C. reports employment at the University of North Carolina-Chapel Hill, Lineberger Cancer Institute. K.M. reports employment at the College of Nursing, University of Utah. K.T. reports employment at the Huntsman Cancer Institute, University of Utah.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"117"}}