{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T06:21:56Z","timestamp":1778307716706,"version":"3.51.4"},"reference-count":37,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,11,2]],"date-time":"2023-11-02T00:00:00Z","timestamp":1698883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Accurately predicting patient outcomes is crucial for improving healthcare delivery, but large-scale risk prediction models are often developed and tested on specific datasets where clinical parameters and outcomes may not fully reflect local clinical settings. Where this is the case, whether to opt for de-novo training of prediction models on local datasets, direct porting of externally trained models, or a transfer learning approach is not well studied, and constitutes the focus of this study. Using the clinical challenge of predicting mortality and hospital length of stay on a Danish trauma dataset, we hypothesized that a transfer learning approach of models trained on large external datasets would provide optimal prediction results compared to de-novo training on sparse but local datasets or directly porting externally trained models.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Using an external dataset of trauma patients from the US Trauma Quality Improvement Program (TQIP) and a local dataset aggregated from the Danish Trauma Database (DTD) enriched with Electronic Health Record data, we tested a range of model-level approaches focused on predicting trauma mortality and hospital length of stay on DTD data. Modeling approaches included de-novo training of models on DTD data, direct porting of models trained on TQIP data to the DTD, and a transfer learning approach by training a model on TQIP data with subsequent transfer and retraining on DTD data. Furthermore, data-level approaches, including mixed dataset training and methods countering imbalanced outcomes (e.g., low mortality rates), were also tested.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Using a neural network trained on a mixed dataset consisting of a subset of TQIP and DTD, with class weighting and transfer learning (retraining on DTD), we achieved excellent results in predicting mortality, with a ROC-AUC of 0.988 and an F2-score of 0.866. The best-performing models for predicting long-term hospitalization were trained only on local data, achieving an ROC-AUC of 0.890 and an F1-score of 0.897, although only marginally better than alternative approaches.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Our results suggest that when assessing the optimal modeling approach, it is important to have domain knowledge of how incidence rates and workflows compare between hospital systems and datasets where models are trained. Including data from other health-care systems is particularly beneficial when outcomes are suffering from class imbalance and low incidence. Scenarios where outcomes are not directly comparable are best addressed through either de-novo local training or a transfer learning approach.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdgth.2023.1249258","type":"journal-article","created":{"date-parts":[[2023,11,3]],"date-time":"2023-11-03T05:54:58Z","timestamp":1698990898000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Assessing optimal methods for transferring machine learning models to low-volume and imbalanced clinical datasets: experiences from predicting outcomes of Danish trauma patients"],"prefix":"10.3389","volume":"5","author":[{"given":"Andreas Skov","family":"Millarch","sequence":"first","affiliation":[]},{"given":"Alexander","family":"Bonde","sequence":"additional","affiliation":[]},{"given":"Mikkel","family":"Bonde","sequence":"additional","affiliation":[]},{"given":"Kiril Vadomovic","family":"Klein","sequence":"additional","affiliation":[]},{"given":"Fredrik","family":"Folke","sequence":"additional","affiliation":[]},{"given":"S\u00f8ren Steemann","family":"Rudolph","sequence":"additional","affiliation":[]},{"given":"Martin","family":"Sillesen","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,11,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"939","DOI":"10.1097\/ALN.0000000000003223","article-title":"Machine learning comes of age","volume":"132","author":"Burns","year":"2020","journal-title":"Anesthesiology"},{"key":"B2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1136\/injuryprev-2015-041616","article-title":"The global burden of injury: incidence, mortality, disability-adjusted life years and time trends from the global burden of disease study 2013","volume":"22","author":"Haagsma","year":"2016","journal-title":"Inj Prev"},{"key":"B3","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1097\/TA.0000000000003158","article-title":"Trauma outcome predictor: an artificial intelligence interactive smartphone tool to predict outcomes in trauma patients","volume":"91","author":"Maurer","year":"2021","journal-title":"J Trauma Acute Care Surg"},{"key":"B4","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.injury.2010.08.040","article-title":"The trauma and injury severity score (TRISS) revised","volume":"42","author":"Schluter","year":"2011","journal-title":"Injury"},{"key":"B5","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1016\/j.injury.2016.12.009","article-title":"Mortality prediction models in the general trauma population: a systematic review","volume":"48","author":"de Munter","year":"2017","journal-title":"Injury"},{"key":"B6","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1111\/aas.12256","article-title":"Norwegian survival prediction model in trauma: modelling effects of anatomic injury, acute physiology, age, and co-morbidity","volume":"58","author":"Jones","year":"2014","journal-title":"Acta Anaesthesiol Scand"},{"key":"B7","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1016\/j.jamcollsurg.2013.07.385","article-title":"Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons","volume":"217","author":"Bilimoria","year":"2013","journal-title":"J Am Coll Surg"},{"key":"B8","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1111\/aas.12592","article-title":"External validation of the Norwegian survival prediction model in trauma after major trauma in southern Finland","volume":"60","author":"Raj","year":"2016","journal-title":"Acta Anaesthesiol Scand"},{"key":"B9","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1016\/j.jgo.2022.04.004","article-title":"Validation of the ACS NSQIP surgical risk calculator in older patients with colorectal cancer undergoing elective surgery","volume":"13","author":"van der Hulst","year":"2022","journal-title":"J Geriatr Oncol"},{"key":"B10","doi-asserted-by":"publisher","first-page":"e8416","DOI":"10.1097\/MD.0000000000008416","article-title":"Predictive validity of the ACS-NSQIP surgical risk calculator in geriatric patients undergoing lumbar surgery","volume":"96","author":"Wang","year":"2017","journal-title":"Medicine (Baltimore)"},{"key":"B11","doi-asserted-by":"publisher","first-page":"17051","DOI":"10.1038\/s41598-021-95533-2","article-title":"A real-world demonstration of machine learning generalizability in the detection of intracranial hemorrhage on head computerized tomography","volume":"11","author":"Salehinejad","year":"2021","journal-title":"Sci Rep"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e471","DOI":"10.1016\/S2589-7500(21)00084-4","article-title":"Assessing the utility of deep neural networks in predicting postoperative surgical complications: a retrospective study","volume":"3","author":"Bonde","year":"2021","journal-title":"Lancet Digit Health"},{"key":"B13","doi-asserted-by":"publisher","first-page":"e000667","DOI":"10.1136\/tsaco-2020-000667","article-title":"Assessment of post-trauma complications in eight million trauma cases over a decade in the USA","volume":"6","author":"Jakobsen","year":"2021","journal-title":"Trauma Surg Acute Care Open"},{"key":"B14","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1186\/s40537-019-0192-5","article-title":"Survey on deep learning with class imbalance","volume":"6","author":"Johnson","year":"2019","journal-title":"J Big Data"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00554-w","article-title":"Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00537-x","article-title":"Prediction across healthcare settings: a case study in predicting emergency department disposition","volume":"4","author":"Barak-Corren","year":"2021","journal-title":"NPJ Digit Med"},{"key":"B17","doi-asserted-by":"publisher","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann Intern Med"},{"key":"B18","doi-asserted-by":"publisher","first-page":"e025611","DOI":"10.1136\/bmjopen-2018-025611","article-title":"Uniformity in measuring adherence to reporting guidelines: the example of TRIPOD for assessing completeness of reporting of prediction model studies","volume":"9","author":"Heus","year":"2019","journal-title":"BMJ Open"},{"key":"B19","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1186\/s13049-019-0676-5","article-title":"The danish prehospital emergency healthcare system and research possibilities","volume":"27","author":"Lindskou","year":"2019","journal-title":"Scand J Trauma Resusc Emerg Med"},{"key":"B20","year":""},{"key":"B21","year":"2016"},{"key":"B22","author":"Joffe","year":"2021"},{"key":"B23","author":"Pedregosa","year":""},{"key":"B24","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/978-3-642-41136-6_5","article-title":"Explaining adaBoost","volume-title":"Empirical inference: Festschrift in honor of Vladimir N. Vapnik","author":"Schapire","year":"2013"},{"key":"B25","first-page":"785","volume-title":".","author":"Chen","year":"2016"},{"key":"B26","author":"Bent\u00e9jac","year":"2019"},{"key":"B27","author":"Nori","year":"2019"},{"key":"B28","doi-asserted-by":"publisher","first-page":"108","DOI":"10.3390\/info11020108","article-title":"Fastai: a layered API for deep learning","volume":"11","author":"Howard","year":"2020","journal-title":"Information"},{"key":"B29","author":"Paszke","year":""},{"key":"B30","doi-asserted-by":"publisher","first-page":"012172","DOI":"10.1088\/1742-6596\/1574\/1\/012172","article-title":"Prediction of poor Students\u2019 classification based on adaboost algorithm integrated learning model","volume":"1574","author":"Li","year":"2020","journal-title":"J Phys Conf Ser"},{"key":"B31","first-page":"233","author":"Davis","year":"2006"},{"key":"B32","doi-asserted-by":"publisher","first-page":"2","DOI":"10.48550\/arXiv.2010.16061","article-title":"Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation","volume":"2","author":"Powers","year":"2008","journal-title":"Mach Learn Technol"},{"key":"B33","author":"Lundberg","year":""},{"key":"B34","first-page":"1","article-title":"Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning","volume":"18","author":"Lema\u00eetre","year":"2017","journal-title":"J Mach Learn Res"},{"key":"B35","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1002\/bjs.11306","article-title":"Validation of the Norwegian survival prediction model in trauma (NORMIT) in Swedish trauma populations","volume":"107","author":"Ghorbani","year":"2020","journal-title":"Br J Surg"},{"key":"B36","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/538020a","article-title":"Can we open the black box of AI?","volume":"538","author":"Ghorbani","year":"2016","journal-title":"Nature News"},{"key":"B37","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1007\/s42452-021-04148-9","article-title":"Comparison of feature importance measures as explanations for classification models","volume":"3","author":"Saarela","year":"2021","journal-title":"SN Appl Sci"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2023.1249258\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,3]],"date-time":"2023-11-03T05:55:11Z","timestamp":1698990911000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2023.1249258\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,2]]},"references-count":37,"alternative-id":["10.3389\/fdgth.2023.1249258"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2023.1249258","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,2]]},"article-number":"1249258"}}