{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T00:15:15Z","timestamp":1786666515518,"version":"3.56.0"},"reference-count":16,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T00:00:00Z","timestamp":1693872000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,12,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Introduction<\/jats:title>\n                  <jats:p>The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying \u201cno time machine rule.\u201d<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Framework<\/jats:title>\n                  <jats:p>We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Recommendations<\/jats:title>\n                  <jats:p>Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad178","type":"journal-article","created":{"date-parts":[[2023,9,5]],"date-time":"2023-09-05T16:30:01Z","timestamp":1693931401000},"page":"274-280","source":"Crossref","is-referenced-by-count":29,"title":["A framework for understanding label leakage in machine learning for health care"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0792-8867","authenticated-orcid":false,"given":"Sharon E","family":"Davis","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael E","family":"Matheny","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37232, United States"},{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37232, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"},{"name":"Tennessee Valley Healthcare System VA Medical Center, Veterans Health Administration , Nashville, TN 37232, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suresh","family":"Balu","sequence":"additional","affiliation":[{"name":"Duke Institute for Health Innovation, Duke University School of Medicine , Durham, NC 27701, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5828-4497","authenticated-orcid":false,"given":"Mark P","family":"Sendak","sequence":"additional","affiliation":[{"name":"Duke Institute for Health Innovation, Duke University School of Medicine , Durham, NC 27701, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,9,5]]},"reference":[{"key":"2023122220304392300_ocad178-B1","volume-title":"Artificial Intelligence in Healthcare: The Hope, the Hype, the Promise, the Peril","author":"Michael","year":"2019"},{"issue":"1","key":"2023122220304392300_ocad178-B2","first-page":"1","article-title":"Machine learning in health care: a critical appraisal of challenges and opportunities","volume":"7","author":"Sendak","year":"2019","journal-title":"EGEMS (Wash DC)"},{"issue":"4","key":"2023122220304392300_ocad178-B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining: formulation, detection, and avoidance","volume":"6","author":"Kaufman","year":"2012","journal-title":"ACM Trans Knowl Discov Data"},{"issue":"2","key":"2023122220304392300_ocad178-B4","doi-asserted-by":"crossref","first-page":"e10969","DOI":"10.2196\/10969","article-title":"Data leakage in health outcomes prediction with machine learning. comment on \u201cprediction of incident hypertension within the next year: prospective study using statewide electronic health records and machine learning\u201d","volume":"23","author":"Chiavegatto Filho","year":"2021","journal-title":"J Med Internet Res"},{"issue":"2","key":"2023122220304392300_ocad178-B5","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1093\/jamiaopen\/ooaa006","article-title":"Machine learning for early detection of sepsis: an internal and temporal validation study","volume":"3","author":"Bedoya","year":"2020","journal-title":"JAMIA Open"},{"key":"2023122220304392300_ocad178-B6","first-page":"191","article-title":"A review of challenges and opportunities in machine learning for health","volume":"2020","author":"Ghassemi","year":"2020","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"issue":"11","key":"2023122220304392300_ocad178-B7","doi-asserted-by":"crossref","first-page":"e1002683","DOI":"10.1371\/journal.pmed.1002683","article-title":"Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study","volume":"15","author":"Zech","year":"2018","journal-title":"PLoS Med"},{"key":"2023122220304392300_ocad178-B8","doi-asserted-by":"crossref","DOI":"10.17226\/21794","volume-title":"Improving Diagnosis in Health Care","author":"Balogh","year":"2015"},{"issue":"11","key":"2023122220304392300_ocad178-B9","doi-asserted-by":"crossref","first-page":"e1002701","DOI":"10.1371\/journal.pmed.1002701","article-title":"Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): a retrospective, single-site study","volume":"15","author":"Corey","year":"2018","journal-title":"PLoS Med"},{"key":"2023122220304392300_ocad178-B10","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1145\/3368555.3384468","article-title":"Hidden stratification causes clinically meaningful failures in machine learning for medical imaging","volume":"2020","author":"Oakden-Rayner","year":"2020","journal-title":"Proc ACM Conf Health Inference Learn (2020)"},{"issue":"24","key":"2023122220304392300_ocad178-B11","doi-asserted-by":"crossref","first-page":"2467","DOI":"10.1001\/jama.2021.22396","article-title":"Next-generation artificial intelligence for diagnosis: from predicting diagnostic labels to \u201cwayfinding\u201d","volume":"326","author":"Adler-Milstein","year":"2021","journal-title":"JAMA"},{"issue":"2","key":"2023122220304392300_ocad178-B12","doi-asserted-by":"crossref","first-page":"e1920733","DOI":"10.1001\/jamanetworkopen.2019.20733","article-title":"Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission","volume":"3","author":"Brajer","year":"2020","journal-title":"JAMA Netw Open"},{"issue":"7","key":"2023122220304392300_ocad178-B13","doi-asserted-by":"crossref","first-page":"e15182","DOI":"10.2196\/15182","article-title":"Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study","volume":"8","author":"Sendak","year":"2020","journal-title":"JMIR Med Inform"},{"issue":"22","key":"2023122220304392300_ocad178-B14","doi-asserted-by":"crossref","first-page":"1489","DOI":"10.1056\/NEJM198906013202211","article-title":"Our stubborn quest for diagnostic certainty. A cause of excessive testing","volume":"320","author":"Kassirer","year":"1989","journal-title":"N Engl J Med"},{"key":"2023122220304392300_ocad178-B15","volume-title":"Predicting Hospital Admissions and Emergency Department Visits in Patients Receiving Immune Checkpoint Inhibitors","author":"Niederhoffer","year":"2022"},{"key":"2023122220304392300_ocad178-B16","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1038\/s41746-020-0253-3","article-title":"Presenting machine learning model information to clinical end users with model facts labels","volume":"3","author":"Sendak","year":"2020","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/1\/274\/54762187\/ocad178.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/1\/274\/54762187\/ocad178.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,22]],"date-time":"2023-12-22T20:31:21Z","timestamp":1703277081000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/1\/274\/7260636"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,5]]},"references-count":16,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,9,5]]},"published-print":{"date-parts":[[2023,12,22]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad178","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,1,1]]},"published":{"date-parts":[[2023,9,5]]}}}