{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T14:30:39Z","timestamp":1791469839090,"version":"4.3.4"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T00:00:00Z","timestamp":1636934400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T00:00:00Z","timestamp":1636934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012774","name":"Innovationsfonden","doi-asserted-by":"publisher","award":["8053-00076B"],"award-info":[{"award-number":["8053-00076B"]}],"id":[{"id":"10.13039\/100012774","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.<\/jats:p>","DOI":"10.1038\/s41746-021-00529-x","type":"journal-article","created":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T06:03:36Z","timestamp":1636956216000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8823-5047","authenticated-orcid":false,"given":"Simon Meyer","family":"Lauritsen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bo","family":"Thiesson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3932-4822","authenticated-orcid":false,"given":"Marianne Johansson","family":"J\u00f8rgensen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anders Hammerich","family":"Riis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3659-650X","authenticated-orcid":false,"given":"Ulrick Skipper","family":"Espelund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesper Bo","family":"Weile","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeppe","family":"Lange","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,15]]},"reference":[{"key":"529_CR1","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1007\/BF01709751","volume":"22","author":"J-L Vincent","year":"2018","unstructured":"Vincent, J.-L. et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction\/failure. Intens. Care Med. 22, 707\u2013710 (2018).","journal-title":"Intens. Care Med."},{"key":"529_CR2","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1001\/jama.2016.0288","volume":"315","author":"CW Seymour","year":"2016","unstructured":"Seymour, C. W. et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 315, 762\u2013774 (2016).","journal-title":"JAMA"},{"key":"529_CR3","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.mpdhp.2016.06.005","volume":"22","author":"C Kartsonaki","year":"2016","unstructured":"Kartsonaki, C. Survival analysis. Diagn. Histopathol. 22, 263\u2013270 (2016).","journal-title":"Diagn. Histopathol."},{"key":"529_CR4","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1007\/s00134-019-05872-y","volume":"46","author":"LM Fleuren","year":"2020","unstructured":"Fleuren, L. M. et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intens. Care Med. 46, 383\u2013400 (2020).","journal-title":"Intens. Care Med."},{"key":"529_CR5","doi-asserted-by":"publisher","first-page":"617486","DOI":"10.3389\/fmed.2021.617486","volume":"8","author":"DR Giacobbe","year":"2021","unstructured":"Giacobbe, D. R. et al. Early detection of sepsis with machine learning techniques: a brief clinical perspective. Front. Med. 8, 617486 (2021).","journal-title":"Front. Med."},{"key":"529_CR6","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-17431-x","volume":"11","author":"SM Lauritsen","year":"2020","unstructured":"Lauritsen, S. M. et al. Explainable artificial intelligence model to predict acute critical illness from electronic health records. Nat. Commun. 11, 3852 (2020).","journal-title":"Nat. Commun."},{"key":"529_CR7","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.compbiomed.2019.04.027","volume":"109","author":"C Barton","year":"2019","unstructured":"Barton, C. et al. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput Biol. Med. 109, 79\u201384 (2019).","journal-title":"Comput Biol. Med."},{"key":"529_CR8","doi-asserted-by":"publisher","first-page":"103395","DOI":"10.1016\/j.compbiomed.2019.103395","volume":"113","author":"M Scherpf","year":"2019","unstructured":"Scherpf, M., Gr\u00e4\u00dfer, F., Malberg, H. & Zaunseder, S. Predicting sepsis with a recurrent neural network using the MIMIC III database. Comput Biol. Med 113, 103395 (2019).","journal-title":"Comput Biol. Med"},{"key":"529_CR9","unstructured":"Moor, M., Horn, M., Rieck, B., Roqueiro, D. & Borgwardt, K. Temporal convolutional networks and dynamic time warping can drastically improve the early prediction of Sepsis. In Proceedings of the 4th Machine Learning for Healthcare Conference. PMLR 106 (2019)."},{"key":"529_CR10","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"529_CR11","doi-asserted-by":"crossref","unstructured":"Lea, C., Vidal, R., Reiter, A. & Hager, G. D. Temporal convolutional networks: a unified approach to action segmentation. In Computer Vision \u2013 ECCV 2016 Workshops. 47\u201354 (Springer International Publishing, 2016).","DOI":"10.1007\/978-3-319-49409-8_7"},{"key":"529_CR12","unstructured":"Bai, S., Kolter, J. Z. & Koltun, V. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. Preprint at https:\/\/arxiv.org\/abs\/1803.01271 (2018)."},{"key":"529_CR13","unstructured":"Futoma, J., Hariharan, S. & Heller, K. Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier. in (eds. Precup, D. & Teh, Y. W.) (2017)."},{"key":"529_CR14","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/j.ijmedinf.2018.12.002","volume":"122","author":"Fvan Wyk","year":"2018","unstructured":"Wyk, Fvan et al. A minimal set of physiomarkers in high frequency real-time physiological data streams predict adult sepsis onset earlier. Int. J. Med. Inf. 122, 55\u201362 (2018).","journal-title":"Int. J. Med. Inf."},{"key":"529_CR15","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","volume":"46","author":"S Nemati","year":"2017","unstructured":"Nemati, S. et al. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit. Care Med. 46, 547\u2013553 (2017).","journal-title":"Crit. Care Med."},{"key":"529_CR16","unstructured":"Futoma, J. et al. An improved multi-output gaussian process RNN with real-time validation for early sepsis detection. Proc. Mach. Learn. Healthcare 2017. JMLR, 68 (2017)."},{"key":"529_CR17","doi-asserted-by":"publisher","first-page":"185","DOI":"10.3414\/ME18-01-0014","volume":"57","author":"A Khojandi","year":"2018","unstructured":"Khojandi, A., Tansakul, V., Li, X., Koszalinski, R. & Paiva, W. Prediction of sepsis and in-hospital mortality using electronic health records. Method Inf. Med. 57, 185\u2013193 (2018).","journal-title":"Method Inf. Med."},{"key":"529_CR18","doi-asserted-by":"publisher","unstructured":"Khoshnevisan, F. et al. Recent Temporal Pattern Mining for Septic Shock Early Prediction. 2018 Ieee Int Conf Healthc Informatics Ichi 229\u2013240 (2018) https:\/\/doi.org\/10.1109\/ichi.2018.00033.","DOI":"10.1109\/ichi.2018.00033"},{"key":"529_CR19","first-page":"1","volume":"17","author":"J Wiens","year":"2016","unstructured":"Wiens, J., Guttag, J. & Horvitz, E. Patient risk stratification with time-varying parameters: a multitask learning approach. J. Mach. Learn. Res. 17, 1\u201323 (2016).","journal-title":"J. Mach. Learn. Res."},{"key":"529_CR20","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-21390-2","volume":"12","author":"W Yuan","year":"2021","unstructured":"Yuan, W. et al. Temporal bias in case-control design: preventing reliable predictions of the future. Nat. Commun. 12, 1107 (2021).","journal-title":"Nat. Commun."},{"key":"529_CR21","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1093\/jamia\/ocz228","volume":"27","author":"Y Huang","year":"2020","unstructured":"Huang, Y., Li, W., Macheret, F., Gabriel, R. A. & Ohno-Machado, L. A tutorial on calibration measurements and calibration models for clinical prediction models. J. Am. Med Inf. Assn 27, 621\u2013633 (2020).","journal-title":"J. Am. Med Inf. Assn"},{"key":"529_CR22","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1007\/s10654-018-0403-y","volume":"33","author":"S Licher","year":"2018","unstructured":"Licher, S. et al. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study. Eur. J. Epidemiol. 33, 645\u2013655 (2018).","journal-title":"Eur. J. Epidemiol."},{"key":"529_CR23","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1093\/jamiaopen\/ooaa006","volume":"3","author":"AD Bedoya","year":"2020","unstructured":"Bedoya, A. D. et al. Machine learning for early detection of sepsis: an internal and temporal validation study. Jamia Open 3, 252\u2013260 (2020).","journal-title":"Jamia Open"},{"key":"529_CR24","doi-asserted-by":"publisher","unstructured":"Lauritsen, S. M. et al. Early detection of sepsis utilizing deep learning on electronic health record event sequences. Artif. Intell. Med. 101820 (2020) https:\/\/doi.org\/10.1016\/j.artmed.2020.101820.","DOI":"10.1016\/j.artmed.2020.101820"},{"key":"529_CR25","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","volume":"181","author":"A Wong","year":"2021","unstructured":"Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 181, 1065\u20131070 (2021).","journal-title":"JAMA Intern. Med"},{"key":"529_CR26","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1038\/s41591-020-0789-4","volume":"26","author":"SL Hyland","year":"2020","unstructured":"Hyland, S. L. et al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nat. Med. 26, 364\u2013373 (2020).","journal-title":"Nat. Med."},{"key":"529_CR27","doi-asserted-by":"publisher","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito, T. & Rehmsmeier, M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. Plos One 10, e0118432 (2015).","journal-title":"Plos One"},{"key":"529_CR28","doi-asserted-by":"publisher","first-page":"e039996","DOI":"10.1136\/bmjopen-2020-039996","volume":"10","author":"AH Riis","year":"2020","unstructured":"Riis, A. H. et al. Cohort profile: CROSS-TRACKS: a population-based open cohort across healthcare sectors in Denmark. Bmj Open 10, e039996 (2020).","journal-title":"Bmj Open"},{"key":"529_CR29","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1177\/1403494810387965","volume":"39","author":"CB Pedersen","year":"2011","unstructured":"Pedersen, C. B. The Danish civil registration system. Scand. J. Public Heal. 39, 22\u201325 (2011).","journal-title":"Scand. J. Public Heal."},{"key":"529_CR30","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1001\/jama.2016.0287","volume":"315","author":"M Singer","year":"2016","unstructured":"Singer, M. et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). Jama 315, 801\u2013810 (2016).","journal-title":"Jama"},{"key":"529_CR31","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","volume":"63","author":"P Geurts","year":"2006","unstructured":"Geurts, P., Ernst, D. & Wehenkel, L. Extremely randomized trees. Mach. Learn 63, 3\u201342 (2006).","journal-title":"Mach. Learn"},{"key":"529_CR32","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L. Random forests. Mach. Learn 45, 5\u201332 (2001).","journal-title":"Mach. Learn"},{"key":"529_CR33","unstructured":"Ke, G. et al. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. in Advances in Neural Information Processing Systems vol. 30 (Curran Associates, Inc., 2017)."},{"key":"529_CR34","doi-asserted-by":"crossref","unstructured":"Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785\u2013794 (2016).","DOI":"10.1145\/2939672.2939785"},{"key":"529_CR35","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T. & Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359\u2013378 (2007).","journal-title":"J. Am. Stat. Assoc."},{"key":"529_CR36","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s10115-013-0670-6","volume":"41","author":"BC Wallace","year":"2014","unstructured":"Wallace, B. C. & Dahabreh, I. J. Improving class probability estimates for imbalanced data. Knowl. Inf. Syst. 41, 33\u201352 (2014).","journal-title":"Knowl. Inf. Syst."},{"key":"529_CR37","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1016\/j.jclinepi.2009.11.009","volume":"63","author":"K Rufibach","year":"2010","unstructured":"Rufibach, K. Use of Brier score to assess binary predictions. J. Clin. Epidemiol. 63, 938\u2013939 (2010).","journal-title":"J. Clin. Epidemiol."},{"key":"529_CR38","doi-asserted-by":"crossref","unstructured":"Zhang, X. Meng, D. Gouk, H. & Hospedales, T. Shallow bayesian meta learning for real-world few-shot recognition. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV). 651\u2013660 (2021).","DOI":"10.1109\/ICCV48922.2021.00069"},{"key":"529_CR39","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","volume":"2","author":"SM Lundberg","year":"2020","unstructured":"Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2, 56\u201367 (2020).","journal-title":"Nat. Mach. Intell."},{"key":"529_CR40","first-page":"4765","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30, 4765\u20134774 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00529-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00529-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00529-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T14:03:11Z","timestamp":1670076191000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00529-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,15]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["529"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00529-x","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,15]]},"assertion":[{"value":"14 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"All authors have completed the ICMJE uniform disclosure form at\n                      \n                      and declare that S.M.L, A.H.R, and B.T. are employed at Enversion A\/S. The authors have no other financial and non-financial interests to disclose.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"158"}}