{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T04:55:03Z","timestamp":1783400103123,"version":"3.54.6"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","issue":"S12","license":[{"start":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T00:00:00Z","timestamp":1664496000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T00:00:00Z","timestamp":1664496000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000886","name":"Duke Endowment","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000886","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Artificial intelligence (AI) and machine learning (ML) have resulted in significant enthusiasm for their promise in healthcare. Despite this, prospective randomized controlled trials and successful clinical implementation remain limited. One clinical application of ML is mitigation of the increased risk for acute care during outpatient cancer therapy. We previously reported the results of the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) study (NCT04277650), which was a prospective, randomized quality improvement study demonstrating that ML based on electronic health record (EHR) data can direct supplemental clinical evaluations and reduce the rate of acute care during cancer radiotherapy\u00a0with and without chemotherapy. The objective of this study is to report the workflow and operational challenges encountered during ML implementation on the SHIELD-RT study.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Data extraction and manual review steps in the workflow represented significant time commitments for implementation of clinical ML on a prospective, randomized study. Barriers include limited data availability through the standard clinical workflow and commercial products, the need to aggregate data from multiple sources, and logistical challenges from altering the standard clinical workflow to deliver adaptive care.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>The SHIELD-RT study was an early randomized controlled study which enabled assessment of barriers to clinical ML implementation, specifically those which leverage the EHR. These challenges build on a growing body of literature and may provide lessons for future healthcare ML adoption.<\/jats:p>\n                <jats:p><jats:bold>Trial registration<\/jats:bold>: NCT04277650. Registered 20 February 2020. Retrospectively registered quality improvement study.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12859-022-04940-3","type":"journal-article","created":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T14:13:41Z","timestamp":1664547221000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Implementation of machine learning in the clinic: challenges and lessons in prospective deployment from the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) randomized controlled study"],"prefix":"10.1186","volume":"23","clinical-trial-number":[{"clinical-trial-number":"nct04277650","registry":"10.18810\/clinical-trials-gov"}],"author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5172-6889","authenticated-orcid":false,"given":"Julian C.","family":"Hong","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neville C. W.","family":"Eclov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah J.","family":"Stephens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yvonne M.","family":"Mowery","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manisha","family":"Palta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,9,30]]},"reference":[{"key":"4940_CR1","doi-asserted-by":"publisher","first-page":"e16323","DOI":"10.2196\/16323","volume":"21","author":"E Coiera","year":"2019","unstructured":"Coiera E. The last mile: Where artificial intelligence meets reality. J Med Internet Res. 2019;21:e16323.","journal-title":"J Med Internet Res"},{"key":"4940_CR2","doi-asserted-by":"publisher","unstructured":"Beede E, Baylor E, Hersch F, Iurchenko A, Wilcox L, Ruamviboonsuk P, et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In: Proceedings of the 2020 CHI conference on human factors in computing systems. Honolulu, HI, USA: Association for Computing Machinery; 2020. p. 1\u201312. doi:https:\/\/doi.org\/10.1145\/3313831.3376718.","DOI":"10.1145\/3313831.3376718"},{"key":"4940_CR3","doi-asserted-by":"publisher","first-page":"1380","DOI":"10.1038\/s41591-020-1045-7","volume":"26","author":"R Nimri","year":"2020","unstructured":"Nimri R, Battelino T, Laffel LM, Slover RH, Schatz D, Weinzimer SA, et al. Insulin dose optimization using an automated artificial intelligence-based decision support system in youths with type 1 diabetes. Nat Med. 2020;26:1380\u20134.","journal-title":"Nat Med"},{"key":"4940_CR4","doi-asserted-by":"publisher","first-page":"3652","DOI":"10.1200\/JCO.20.01688","volume":"38","author":"JC Hong","year":"2020","unstructured":"Hong JC, Eclov NCW, Dalal NH, Thomas SM, Stephens SJ, Malicki M, et al. System for High-Intensity Evaluation during Radiation Therapy (SHIELD-RT): a prospective randomized study of machine learning-directed clinical evaluations during radiation and chemoradiation. JCO. 2020;38:3652\u201361.","journal-title":"JCO"},{"key":"4940_CR5","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2020.0592","author":"M Wijnberge","year":"2020","unstructured":"Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. JAMA. 2020. https:\/\/doi.org\/10.1001\/jama.2020.0592.","journal-title":"JAMA"},{"key":"4940_CR6","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1377\/hlthaff.2018.05081","volume":"38","author":"K Safavi","year":"2019","unstructured":"Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-funded digital health companies and their impact on high-burden. High-Cost Conditions Health Affairs. 2019;38:115\u201323.","journal-title":"High-Cost Conditions Health Affairs"},{"key":"4940_CR7","doi-asserted-by":"publisher","first-page":"1028","DOI":"10.1001\/jamaoncol.2019.0086","volume":"5","author":"V Jairam","year":"2019","unstructured":"Jairam V, Lee V, Park HS, Thomas CR, Melnick ER, Gross CP, et al. Treatment-related complications of systemic therapy and radiotherapy. JAMA Oncol. 2019;5:1028\u201335.","journal-title":"JAMA Oncol"},{"key":"4940_CR8","doi-asserted-by":"publisher","first-page":"e622","DOI":"10.1200\/JOP.19.00551","volume":"16","author":"CM Phillips","year":"2020","unstructured":"Phillips CM, Deal K, Powis M, Singh S, Dharmakulaseelan L, Naik H, et al. Evaluating patients\u2019 perception of the risk of acute care visits during systemic therapy for cancer. JCO Oncol Pract. 2020;16:e622\u20139.","journal-title":"JCO Oncol Pract."},{"key":"4940_CR9","unstructured":"Admissions and Emergency Department (ED) Visits for Patients Receiving Outpatient Chemotherapy. https:\/\/cmit.cms.gov\/CMIT_public\/ViewMeasure?MeasureId=2929. Accessed 19 Dec 2019."},{"key":"4940_CR10","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1038\/s41591-019-0651-8","volume":"26","author":"KE Morse","year":"2020","unstructured":"Morse KE, Bagley SC, Shah NH. Estimate the hidden deployment cost of predictive models to improve patient care. Nat Med. 2020;26:18\u20139.","journal-title":"Nat Med"},{"key":"4940_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1200\/CCI.18.00037","volume":"2","author":"JC Hong","year":"2018","unstructured":"Hong JC, Niedzwiecki D, Palta M, Tenenbaum JD. Predicting Emergency visits and hospital admissions during radiation and chemoradiation: an internally validated pretreatment machine learning algorithm. JCO Clinical Cancer Informatics. 2018;2:1\u201311.","journal-title":"JCO Clinical Cancer Informatics"},{"key":"4940_CR12","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1016\/j.jbi.2010.11.008","volume":"44","author":"MM Horvath","year":"2011","unstructured":"Horvath MM, Winfield S, Evans S, Slopek S, Shang H, Ferranti J. The DEDUCE Guided Query tool: providing simplified access to clinical data for research and quality improvement. J Biomed Inform. 2011;44:266\u201376.","journal-title":"J Biomed Inform"},{"key":"4940_CR13","first-page":"548","volume":"2007","author":"SN Murphy","year":"2022","unstructured":"Murphy SN, Mendis M, Hackett K, Kuttan R, Pan W, Phillips LC, et al. Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside. AMIA Annu Symp Proc. 2022;2007:548\u201352.","journal-title":"AMIA Annu Symp Proc."},{"key":"4940_CR14","first-page":"552","volume":"2002","author":"SN Murphy","year":"2022","unstructured":"Murphy SN, Chueh HC. A security architecture for query tools used to access large biomedical databases. Proc AMIA Symp. 2022;2002:552\u20136.","journal-title":"Proc AMIA Symp."},{"key":"4940_CR15","first-page":"391","volume":"2009","author":"HJ Lowe","year":"2009","unstructured":"Lowe HJ, Ferris TA, Hernandez PM, Weber SC. STRIDE\u2013An integrated standards-based translational research informatics platform. AMIA Annu Symp Proc. 2009;2009:391\u20135.","journal-title":"AMIA Annu Symp Proc"},{"key":"4940_CR16","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.jbi.2008.08.010","volume":"42","author":"PA Harris","year":"2009","unstructured":"Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap)\u2014a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377\u201381.","journal-title":"J Biomed Inform"},{"key":"4940_CR17","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1093\/jamiaopen\/ooaa064","volume":"3","author":"JC Hong","year":"2020","unstructured":"Hong JC, Fairchild AT, Tanksley JP, Palta M, Tenenbaum JD. Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts. JAMIA Open. 2020;3:513\u20137.","journal-title":"JAMIA Open"},{"key":"4940_CR18","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1093\/jamia\/ocw105","volume":"24","author":"D Lyell","year":"2017","unstructured":"Lyell D, Coiera E. Automation bias and verification complexity: a systematic review. J Am Med Inform Assoc. 2017;24:423\u201331.","journal-title":"J Am Med Inform Assoc"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-022-04940-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-022-04940-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-022-04940-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T14:14:33Z","timestamp":1664547273000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-022-04940-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,30]]},"references-count":18,"journal-issue":{"issue":"S12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["4940"],"URL":"https:\/\/doi.org\/10.1186\/s12859-022-04940-3","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,30]]},"assertion":[{"value":"14 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 September 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study was approved by the Duke University Medical Center Institutional Review Board (Pro00100647) as a quality improvement study. As a quality improvement study, study consent was not required.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"NA.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"JCH and MP, are coinventors on a pending patent, \u201cSystems and methods for predicting acute care visits during outpatient cancer therapy,\u201d studied in this manuscript.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"408"}}