{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:52:58Z","timestamp":1787032378479,"version":"build-2736575974"},"reference-count":33,"publisher":"Harborside Press, LLC","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:p>\n                    <jats:bold>Background:<\/jats:bold>\n                    Emergency department visits and hospitalizations frequently occur during systemic therapy for cancer. We developed and evaluated a longitudinal warning system for acute care use.\n                    <jats:bold>Methods:<\/jats:bold>\n                    Using a retrospective population-based cohort of patients who started intravenous systemic therapy for nonhematologic cancers between July 1, 2014, and June 30, 2020, we randomly separated patients into cohorts for model training, hyperparameter tuning and model selection, and system testing. Predictive features included static features, such as demographics, cancer type, and treatment regimens, and dynamic features, such as patient-reported symptoms and laboratory values. The longitudinal warning system predicted the probability of acute care utilization within 30 days after each treatment session. Machine learning systems were developed in the training and tuning cohorts and evaluated in the testing cohort. Sensitivity analyses considered feature importance, other acute care endpoints, and performance within subgroups.\n                    <jats:bold>Results:<\/jats:bold>\n                    The cohort included 105,129 patients who received 1,216,385 treatment sessions. Acute care followed 182,444 (15.0%) treatments within 30 days. The ensemble model achieved an area under the receiver operating characteristic curve of 0.742 (95% CI, 0.739\u20130.745) and was well calibrated in the test cohort. Important predictive features included prior acute care use, treatment regimen, and laboratory tests. If the system was set to alarm approximately once every 15 treatments, 25.5% of acute care events would be preceded by an alarm, and 47.4% of patients would experience acute care after an alarm. The system underestimated risk for some treatment regimens and potentially underserved populations such as females and non-English speakers.\n                    <jats:bold>Conclusions:<\/jats:bold>\n                    Machine learning warning systems can detect patients at risk for acute care utilization, which can aid in preventive intervention and facilitate tailored treatment. Future research should address potential biases and prospectively evaluate impact after system deployment.\n                  <\/jats:p>","DOI":"10.6004\/jnccn.2023.7046","type":"journal-article","created":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T15:27:22Z","timestamp":1697124442000},"page":"1029-1037.e21","source":"Crossref","is-referenced-by-count":3,"title":["Machine Learning\u2013Based Early Warning Systems for Acute Care Utilization During Systemic Therapy for Cancer"],"prefix":"10.6004","volume":"21","author":[{"given":"Robert C.","family":"Grant","sequence":"first","affiliation":[{"name":"Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada"},{"name":"Ontario Institute for Cancer Research, Toronto, Ontario, Canada"},{"name":"Vector Institute, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiang Chen","family":"He","sequence":"additional","affiliation":[{"name":"Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada"},{"name":"Ontario Institute for Cancer Research, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ferhana","family":"Khan","sequence":"additional","affiliation":[{"name":"ICES, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ning","family":"Liu","sequence":"additional","affiliation":[{"name":"ICES, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sho","family":"Podolsky","sequence":"additional","affiliation":[{"name":"ICES, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yosuf","family":"Kaliwal","sequence":"additional","affiliation":[{"name":"ICES, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melanie","family":"Powis","sequence":"additional","affiliation":[{"name":"Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faiyaz","family":"Notta","sequence":"additional","affiliation":[{"name":"Ontario Institute for Cancer Research, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kelvin K.W.","family":"Chan","sequence":"additional","affiliation":[{"name":"ICES, Toronto, Ontario, Canada"},{"name":"Odette Cancer Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marzyeh","family":"Ghassemi","sequence":"additional","affiliation":[{"name":"Vector Institute, Toronto, Ontario, Canada"},{"name":"Massachusetts Institute of Technology, Cambridge, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven","family":"Gallinger","sequence":"additional","affiliation":[{"name":"Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada"},{"name":"Ontario Institute for Cancer Research, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Monika K.","family":"Krzyzanowska","sequence":"additional","affiliation":[{"name":"Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada"},{"name":"ICES, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4047","reference":[{"key":"B1","series-title":"Eur J Cancer Care (Engl)","first-page":"e12909","article-title":"Hospitalisations and emergency department visits in cancer patients receiving systemic therapy: systematic review and meta-analysis","volume":"28","author":"Prince RM","year":"2019","unstructured":"Prince RM, Powis M, Zer A,  Hospitalisations and emergency department visits in cancer patients receiving systemic therapy: systematic review and meta-analysis. Eur J Cancer Care (Engl) 2019;28:e12909."},{"key":"B2","series-title":"Health Aff (Millwood)","first-page":"1793","article-title":"Acute hospital care is the chief driver of regional spending variation in Medicare patients with advanced cancer","volume":"33","author":"Brooks GA","year":"2014","unstructured":"Brooks GA, Li L, Uno H,  Acute hospital care is the chief driver of regional spending variation in Medicare patients with advanced cancer. Health Aff (Millwood) 2014;33:1793\u20131800."},{"key":"B3","series-title":"CMAJ Open","first-page":"E1","article-title":"The economic burden of cancer care in Canada: a population-based cost study","volume":"6","author":"de Oliveira C","year":"2018","unstructured":"de Oliveira C, Weir S, Rangrej J,  The economic burden of cancer care in Canada: a population-based cost study. CMAJ Open 2018;6:E1\u201310."},{"key":"B4","series-title":"Lancet Oncol","first-page":"1165","article-title":"Economic burden of cancer across the European Union: a population-based cost analysis","volume":"14","author":"Luengo-Fernandez R","year":"2013","unstructured":"Luengo-Fernandez R, Leal J, Gray A,  Economic burden of cancer across the European Union: a population-based cost analysis. Lancet Oncol 2013;14:1165\u20131174."},{"key":"B5","article-title":"Systemic treatment provincial plan 2014\u20132019","unstructured":"Cancer Care Ontario. Systemic treatment provincial plan 2014\u20132019. Accessed May 15, 2023. Available at: https:\/\/www.cancercareontario.ca\/en\/cancer-care-ontario\/programs\/clinical-services\/systemic-treatment\/provincial-plan"},{"key":"B6","article-title":"Hospital outpatient quality reporting program","unstructured":"Centers for Medicare & Medicaid Services. Hospital outpatient quality reporting program. Accessed May 15, 2023. Available at: https:\/\/www.cms.gov\/medicare\/quality-initiatives-patient-assessment-instruments\/hospitalqualityinits\/hospitaloutpatientqualityreportingprogram"},{"key":"B7","series-title":"N Engl J Med","first-page":"486","article-title":"Automated identification of adults at risk for in-hospital clinical deterioration. Reply","volume":"384","author":"Escobar GJ","year":"2021","unstructured":"Escobar GJ, Liu VX, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. Reply. N Engl J Med 2021;384:486."},{"key":"B8","series-title":"JAMA","first-page":"1052","article-title":"Effect of a machine learning- derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial","volume":"323","author":"Wijnberge M","year":"2020","unstructured":"Wijnberge M, Geerts BF, Hol L,  Effect of a machine learning- derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. JAMA 2020;323:1052\u20131060."},{"key":"B9","series-title":"JAMA Netw Open","first-page":"e1912823","article-title":"Development and validation of a score to predict acute care use after initiation of systemic therapy for cancer","volume":"2","author":"Grant RC","year":"2019","unstructured":"Grant RC, Moineddin R, Yao Z,  Development and validation of a score to predict acute care use after initiation of systemic therapy for cancer. JAMA Netw Open 2019;2:e1912823."},{"key":"B10","series-title":"JCO Clin Cancer Inform","first-page":"266","article-title":"Development of an oncology acute care risk prediction model","volume":"5","author":"Csik VP","year":"2021","unstructured":"Csik VP, Li M, Binder AF,  Development of an oncology acute care risk prediction model. JCO Clin Cancer Inform 2021;5:266\u2013271."},{"key":"B11","series-title":"JCO Clin Cancer Inform","first-page":"1106","article-title":"Machine learning applied to electronic health records: identification of chemotherapy patients at high risk for preventable emergency department visits and hospital admissions","volume":"5","author":"Peterson DJ","year":"2021","unstructured":"Peterson DJ, Ostberg NP, Blayney DW,  Machine learning applied to electronic health records: identification of chemotherapy patients at high risk for preventable emergency department visits and hospital admissions. JCO Clin Cancer Inform 2021;5:1106\u20131126."},{"key":"B12","series-title":"J Clin Oncol","first-page":"496","article-title":"Identification of potentially avoidable hospitalizations in patients with GI cancer","volume":"32","author":"Brooks GA","year":"2014","unstructured":"Brooks GA, Abrams TA, Meyerhardt JA,  Identification of potentially avoidable hospitalizations in patients with GI cancer. J Clin Oncol 2014;32:496\u2013503."},{"key":"B13","series-title":"JAMA Oncol","first-page":"441","article-title":"A clinical prediction model to assess risk for chemotherapy-related hospitalization in patients initiating palliative chemotherapy","volume":"1","author":"Brooks GA","year":"2015","unstructured":"Brooks GA, Kansagra AJ, Rao SR,  A clinical prediction model to assess risk for chemotherapy-related hospitalization in patients initiating palliative chemotherapy. JAMA Oncol 2015;1:441\u2013447."},{"key":"B14","series-title":"Ann Intern Med","first-page":"W1","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons KG","year":"2015","unstructured":"Moons KG, Altman DG, Reitsma JB,  Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1\u201373."},{"key":"B15","series-title":"Int J Popul Data Sci","first-page":"1135","article-title":"ICES: data, discovery, better health","volume":"4","author":"Schull MJ","year":"2020","unstructured":"Schull MJ, Azimaee M, Marra M,  ICES: data, discovery, better health. Int J Popul Data Sci 2020;4:1135."},{"key":"B16","series-title":"BMC Med Inform Decis Mak","first-page":"135","article-title":"Describing the linkages of the Immigration, Refugees and Citizenship Canada permanent resident data and vital statistics death registry to Ontario\u2019s administrative health database","volume":"16","author":"Chiu M","year":"2016","unstructured":"Chiu M, Lebenbaum M, Lam K,  Describing the linkages of the Immigration, Refugees and Citizenship Canada permanent resident data and vital statistics death registry to Ontario\u2019s administrative health database. BMC Med Inform Decis Mak 2016;16:135."},{"key":"B17","series-title":"J Pain Symptom Manage","first-page":"456","article-title":"A multicenter study comparing two numerical versions of the Edmonton Symptom Assessment System in palliative care patients","volume":"41","author":"Watanabe SM","year":"2011","unstructured":"Watanabe SM, Nekolaichuk C, Beaumont C,  A multicenter study comparing two numerical versions of the Edmonton Symptom Assessment System in palliative care patients. J Pain Symptom Manage 2011;41:456\u2013468."},{"key":"B18","series-title":"JPEN J Parenter Enteral Nutr","first-page":"915","article-title":"Validation of the scored Patient-Generated Subjective Global Assessment Short Form as a prognostic tool for patients with incurable cancer","volume":"46","author":"Cunha MS","year":"2022","unstructured":"Cunha MS, Wiegert EVM, Calixto-Lima L,  Validation of the scored Patient-Generated Subjective Global Assessment Short Form as a prognostic tool for patients with incurable cancer. JPEN J Parenter Enteral Nutr 2022;46:915\u2013922."},{"key":"B19","series-title":"JCO Clin Cancer Inform","first-page":"1","article-title":"Hospitalization risk during chemotherapy for advanced cancer: development and validation of risk stratification models using real-world data","volume":"3","author":"Brooks GA","year":"2019","unstructured":"Brooks GA, Uno H, Aiello Bowles EJ,  Hospitalization risk during chemotherapy for advanced cancer: development and validation of risk stratification models using real-world data. JCO Clin Cancer Inform 2019;3:1\u201310."},{"key":"B20","series-title":"J Oncol Pract","first-page":"e51","article-title":"Can chemotherapy- related acute care visits be accurately identified in administrative data?","volume":"14","author":"Krzyzanowska MK","year":"2018","unstructured":"Krzyzanowska MK, Enright K, Moineddin R,  Can chemotherapy- related acute care visits be accurately identified in administrative data? J Oncol Pract 2018;14:e51\u201358."},{"key":"B21","series-title":"BMC Med","first-page":"230","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster B","year":"2019","unstructured":"Van Calster B, McLernon DJ, van Smeden M,  Calibration: the Achilles heel of predictive analytics. BMC Med 2019;17:230."},{"key":"B22","series-title":"Stat Med","first-page":"4051","article-title":"The integrated calibration index (ICI) and related metrics for quantifying the calibration of logistic regression models","volume":"38","author":"Austin PC","year":"2019","unstructured":"Austin PC, Steyerberg EW. The integrated calibration index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med 2019;38:4051\u20134065."},{"key":"B23","series-title":"Diagn Progn Res","first-page":"18","article-title":"A simple, step-by-step guide to interpreting decision curve analysis","volume":"3","author":"Vickers AJ","year":"2019","unstructured":"Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 2019;3:18."},{"key":"B24","series-title":"Med Decis Making","first-page":"565","article-title":"Decision curve analysis: a novel method for evaluating prediction models","volume":"26","author":"Vickers AJ","year":"2006","unstructured":"Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565\u2013574."},{"key":"B25","series-title":"CMAJ","first-page":"E1351","article-title":"Implementing machine learning in medicine","volume":"193","author":"Verma AA","year":"2021","unstructured":"Verma AA, Murray J, Greiner R,  Implementing machine learning in medicine. CMAJ 2021;193:E1351\u20131357."},{"key":"B26","series-title":"JCO Clin Cancer Inform","first-page":"275","article-title":"Building a clinically relevant risk model: predicting risk of a potentially preventable acute care visit for patients starting antineoplastic treatment","volume":"4","author":"Daly B","year":"2020","unstructured":"Daly B, Gorenshteyn D, Nicholas KJ,  Building a clinically relevant risk model: predicting risk of a potentially preventable acute care visit for patients starting antineoplastic treatment. JCO Clin Cancer Inform 2020;4:275\u2013289."},{"key":"B27","series-title":"Nat Med","first-page":"2176","article-title":"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations","volume":"27","author":"Seyyed-Kalantari L","year":"2021","unstructured":"Seyyed-Kalantari L, Zhang H, McDermott MBA,  Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med 2021;27:2176\u20132182."},{"key":"B28","series-title":"JCO Clin Cancer Inform","first-page":"839","article-title":"Use of wearable activity tracker in patients with cancer undergoing chemotherapy: toward evaluating risk of unplanned health care encounters","volume":"4","author":"Nilanon T","year":"2020","unstructured":"Nilanon T, Nocera LP, Martin AS,  Use of wearable activity tracker in patients with cancer undergoing chemotherapy: toward evaluating risk of unplanned health care encounters. JCO Clin Cancer Inform 2020;4:839\u2013853."},{"key":"B29","series-title":"J Clin Oncol","first-page":"557","article-title":"Symptom monitoring with patient- reported outcomes during routine cancer treatment: a randomized controlled trial","volume":"34","author":"Basch E","year":"2016","unstructured":"Basch E, Deal AM, Kris MG,  Symptom monitoring with patient- reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016;34:557\u2013565."},{"key":"B30","series-title":"J Biomed Semantics","first-page":"14","article-title":"Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies","volume":"11","author":"Kersloot MG","year":"2020","unstructured":"Kersloot MG, van Putten FJP, Abu-Hanna A,  Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies. J Biomed Semantics 2020;11:14."},{"key":"B31","series-title":"Advances in Neural Information Processing Systems 33","first-page":"6696","article-title":"Neural controlled differential equations for irregular time series","author":"Kidger P","year":"2020","unstructured":"Kidger P, Morrill J, Foster J,  Neural controlled differential equations for irregular time series. In: Lorachell H, Ranzato M, Hadsell R, et al, eds. Advances in Neural Information Processing Systems 33. NeurIPS; 2020:6696\u20136707."},{"key":"B32","series-title":"AACN Adv Crit Care","first-page":"378","article-title":"Alarm fatigue: a patient safety concern","volume":"24","author":"Sendelbach S","year":"2013","unstructured":"Sendelbach S, Funk M. Alarm fatigue: a patient safety concern. AACN Adv Crit Care 2013;24:378\u2013386; quiz 387\u2013388."},{"key":"B33","series-title":"SIGKDD Explor","first-page":"6","article-title":"Machine learning for streaming data: state of the art, challenges, and opportunities","volume":"21","author":"Gomes HM","year":"2019","unstructured":"Gomes HM, Read J, Bifet A,  Machine learning for streaming data: state of the art, challenges, and opportunities. SIGKDD Explor 2019;21:6\u201322."}],"container-title":["Journal of the National Comprehensive Cancer Network"],"original-title":[],"link":[{"URL":"https:\/\/jnccn.org\/view\/journals\/jnccn\/21\/10\/article-p1029.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/jnccn.org\/downloadpdf\/journals\/jnccn\/21\/10\/article-p1029.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jnccn.org\/downloadpdf\/journals\/jnccn\/21\/10\/article-p1029.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T15:27:22Z","timestamp":1697124442000},"score":1,"resource":{"primary":{"URL":"https:\/\/jnccn.org\/view\/journals\/jnccn\/21\/10\/article-p1029.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10]]},"references-count":33,"journal-issue":{"issue":"10"},"URL":"https:\/\/doi.org\/10.6004\/jnccn.2023.7046","relation":{},"ISSN":["1540-1405","1540-1413"],"issn-type":[{"value":"1540-1405","type":"print"},{"value":"1540-1413","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10]]}}}