{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T05:37:41Z","timestamp":1792820261755,"version":"4.5.0"},"reference-count":51,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Cohort selection criteria play a critical role in shaping machine learning (ML) model performance and the equity of clinical outcome predictions across demographic groups. In practice, cohort definitions are often influenced by variable and sometimes inconsistent data processing decisions, which may introduce bias and limit the generalizability of ML models. During the COVID-19 pandemic, rapid cohort construction further increased concerns about transparency and fairness in ML-based analyses.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to systematically examine how cohort selection and data processing decisions influence ML performance and demographic equity in predicting COVID-19\u2013related in-hospital mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Using data from the National COVID Cohort Collaborative (N3C), we evaluated 2 sets of cohorts. Set 1 consisted of 16 cohorts derived from 4 primary data processing decisions, including COVID-19 case identification, inpatient inclusion, diagnosis date selection, and admission timestamp availability. Set 2 expanded this design to 64 cohorts by additionally applying provider ID and location identifier filtering. Model performance was assessed using the area under the receiver operating characteristic curve (AUC) across multiple training-testing cohort combinations. Three ML models\u2014logistic regression, random forest, and gradient boosting\u2014were evaluated using 3 analytical approaches: maximum AUC classification, direct AUC regression, and AUC gap analysis. Performance was further examined across demographic subgroups defined by gender, race, and ethnicity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>This study analyzed data from the N3C, including patients with a first positive COVID-19 diagnosis between August 1, 2020, and December 31, 2021. Data preprocessing, cohort construction, and model development were completed prior to analysis. Cohort selection decisions had a substantial impact on ML model performance. Admission time inclusion or exclusion emerged as the most influential factor in Set 1 and consistently affected model accuracy across analytical approaches. In Set 2, this decision remained important, while additional criteria, particularly provider ID filtering, also significantly influenced results. The importance of specific decisions varied across models and evaluation strategies. Analyses across demographic subgroups showed that data processing decisions affected predictive performance differently by gender, race, and ethnicity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Seemingly minor cohort selection and data processing decisions can meaningfully affect both predictive accuracy and demographic equity in ML-based COVID-19 outcome prediction. These findings highlight the risk of bias introduced by differences in cohort definitions and underscore the need for transparent, standardized, and equity-aware cohort selection practices to support fair and reproducible ML research in health care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/88406","type":"journal-article","created":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T14:52:07Z","timestamp":1787237527000},"page":"e88406-e88406","source":"Crossref","is-referenced-by-count":1,"title":["A Systematic Evaluation of Cohort Selection Criteria and Their Impact on Machine Learning Model Performance and Demographic Disparities in COVID-19 Outcomes: Cohort Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2455-8597","authenticated-orcid":false,"given":"Atefehsadat","family":"Haghighathoseini","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2238-0588","authenticated-orcid":false,"given":"Janusz","family":"Wojtusiak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2422-0043","authenticated-orcid":false,"given":"Hua","family":"Min","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0544-5127","authenticated-orcid":false,"given":"Nirup","family":"M Menon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,20]]},"reference":[{"issue":"14","key":"R1","doi-asserted-by":"publisher","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"Rajkomar","journal-title":"N Engl J Med"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s12884-025-08066-1","article-title":"The development and validation of postpartum hemorrhage prediction models for pregnancies with placenta previa totalis based on coagulation function indexes: a retrospective cohort study","volume":"25","author":"Hong","journal-title":"BMC Pregnancy Childbirth"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2018.04.007","article-title":"Benchmarking deep learning models on large healthcare datasets","volume":"83","author":"Purushotham","journal-title":"J Biomed Inform"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Qui\u00f1onero-Candela J Sugiyama M Schwaighofer A Lawrence ND , editors. Dataset Shift in Machine Learning. MIT Press; 2008. [doi: 10.7551\/mitpress\/9780262170055.001.0001]","DOI":"10.7551\/mitpress\/9780262170055.001.0001"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Guerra-Manzanares A Lopez LJL Maniatakos M Shamout FE . Privacy-preserving machine learning for healthcare: open challenges and future perspectives. In: Trustworthy Machine Learning for Healthcare. Vol 13932. 2023:25-40. [doi: 10.1007\/978-3-031-39539-0_3]","DOI":"10.1007\/978-3-031-39539-0_3"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n2281","article-title":"Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review","volume":"375","author":"Andaur Navarro","journal-title":"BMJ"},{"issue":"6464","key":"R7","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","journal-title":"Science"},{"issue":"11","key":"R8","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000651","article-title":"Bias in medical AI: implications for clinical decision-making","volume":"3","author":"Cross","journal-title":"PLoS Digit Health"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1038\/s41746-019-0101-5","article-title":"Finding missed cases of familial hypercholesterolemia in health systems using machine learning","volume":"2","author":"Banda","journal-title":"NPJ Digit Med"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1048311","article-title":"Prediction of angiogenesis in extrahepatic cholangiocarcinoma using MRI-based machine learning","volume":"13","author":"Liu","journal-title":"Front Oncol"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgast-2019-000342","article-title":"Development of a national department of Veterans Affairs mortality risk prediction model among patients with cirrhosis","volume":"6","author":"Koola","journal-title":"BMJ Open Gastroenterol"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","journal-title":"ACM Comput Surv"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2023.1014773","article-title":"A novel approach for assessing bias during team-based clinical decision-making","volume":"11","author":"Pool","journal-title":"Front Public Health"},{"issue":"6","key":"R14","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1007\/s40615-018-0468-z","article-title":"Factors related to physician clinical decision-making for African-American and Hispanic patients: a qualitative meta-synthesis","volume":"5","author":"Breathett","journal-title":"J Racial Ethn Health Disparities"},{"issue":"7","key":"R15","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.11044","article-title":"Association of gender and race with allocation of advanced heart failure therapies","volume":"3","author":"Breathett","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/j.gastha.2021.11.008","article-title":"Competing risk bias in prognostic models predicting hepatocellular carcinoma occurrence: impact on clinical decision-making","volume":"1","author":"Innes","journal-title":"Gastro Hep Adv"},{"key":"R17","volume":"2024","author":"Haghighathoseini","journal-title":"AMIA Annu Symp Proc"},{"issue":"3","key":"R18","doi-asserted-by":"publisher","first-page":"225","DOI":"10.4997\/JRCPE.2018.306","article-title":"Cognitive bias in clinical medicine","volume":"48","author":"O\u2019Sullivan","journal-title":"J R Coll Physicians Edinb"},{"key":"R19","volume":"2024","author":"Haghighathoseini","journal-title":"AMIA Annu Symp Proc"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2020.570396","article-title":"Computed tomography radiomics for predicting pathological grade of renal cell carcinoma","volume":"10","author":"Yi","journal-title":"Front Oncol"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1007\/s00345-021-03844-w","article-title":"Selection and evaluation of preoperative systemic inflammatory response biomarkers model prior to cytoreductive nephrectomy using a machine-learning approach","volume":"40","author":"Laukhtina","journal-title":"World J Urol"},{"issue":"9","key":"R22","doi-asserted-by":"publisher","first-page":"2435","DOI":"10.3324\/haematol.2022.282370","article-title":"Artificial intelligence to empower diagnosis of myelodysplastic syndromes by multiparametric flow cytometry","volume":"108","author":"Clichet","journal-title":"Haematologica"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Chaganti S Singh V Gent AE Kamaleswaran R Kamen A . Evaluating the impact of common clinical confounders on performance of deep-learning based sepsis risk assessment. In Review. Preprint posted online on Apr 10, 2024. [doi: 10.21203\/rs.3.rs-4017967\/v1]","DOI":"10.21203\/rs.3.rs-4017967\/v1"},{"issue":"7","key":"R24","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics11071155","article-title":"Comprehensive survey of using machine learning in the COVID-19 pandemic","volume":"11","author":"El-Rashidy","journal-title":"Diagnostics (Basel)"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal","volume":"369","author":"Wynants","journal-title":"BMJ"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","first-page":"67","DOI":"10.5811\/westjem.60243","article-title":"Development and external validation of clinical features-based machine learning models for predicting COVID-19 in the emergency department","volume":"25","author":"Tay","journal-title":"West J Emerg Med"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Haghighathoseini A Wojtusiak J Menon NM Min H Frankenfeld C Leslie T . Big data decision-making and racial disparities: a case study among COVID-19 inpatient visits. Presented at: 2024 IEEE International Conference on Big Data (BigData); Dec 15-18, 2024:6452-6459; Washington, DC. [doi: 10.1109\/BigData62323.2024.10825471]","DOI":"10.1109\/BigData62323.2024.10825471"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"52","DOI":"10.22436\/jmcs.029.01.05","article-title":"Modelling the fourth wave of COVID-19 pandemic in Egypt","volume":"29","author":"Moneim","journal-title":"J Math Computer Sci"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Haghighathoseini A Qodrati M Min H et al. Selection Bias from Data Processing in N3C. Presented at: 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI); Jun 3-6, 2024:234-241; Orlando, FL. [doi: 10.1109\/ICHI61247.2024.00038]","DOI":"10.1109\/ICHI61247.2024.00038"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/s12967-023-04387-x","article-title":"A novel explainable online calculator for contrast-induced AKI in diabetics: a multi-centre validation and prospective evaluation study","volume":"21","author":"Ma","journal-title":"J Transl Med"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2024.1430899","article-title":"Development of a COVID-19 early risk assessment system based on multiple machine learning algorithms and routine blood tests: a real-world study","volume":"15","author":"Qin","journal-title":"Front Immunol"},{"key":"#cr-split#-R32.1","doi-asserted-by":"crossref","unstructured":"Chen J Xie Y Wang K et al. Generative invertible networks (GIN): pathophysiology-interpretable feature mapping and virtual patient generation. In: Frangi AF Schnabel JA Davatzikos C Alberola-L\u00f3pez C Fichtinger G , editors. Presented at: Medical Image Computing and Computer Assisted Intervention - MICCAI 2018. Sep 16-20, 2018:Springer. 537-545","DOI":"10.1007\/978-3-030-00928-1_61"},{"key":"#cr-split#-R32.2","doi-asserted-by":"crossref","unstructured":"Granada, Spain. 2018.[doi: 10.1007\/978-3-030-00928-1_61]","DOI":"10.1007\/978-3-030-00928-1_61"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2022.e12681","article-title":"Machine learning predicts cancer-associated venous thromboembolism using clinically available variables in gastric cancer patients","volume":"9","author":"Xu","journal-title":"Heliyon"},{"issue":"7","key":"R34","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-081723","article-title":"Health professionals implicit bias of patients with low socioeconomic status (SES) and its effects on clinical decision-making: a scoping review","volume":"14","author":"Job","journal-title":"BMJ Open"},{"issue":"2","key":"R35","doi-asserted-by":"publisher","first-page":"100324","DOI":"10.1016\/j.gim.2022.10.007","article-title":"COVID-19 in people with neurofibromatosis 1, neurofibromatosis 2, or schwannomatosis","volume":"25","author":"Banerjee","journal-title":"Genet Med"},{"issue":"7","key":"R36","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.16901","article-title":"Clinical characterization and prediction of clinical severity of SARS-CoV-2 infection among US adults using data from the US National COVID cohort collaborative","volume":"4","author":"Bennett","journal-title":"JAMA Netw Open"},{"key":"R37","unstructured":"National Clinical Cohort Collaborative (N3C) Homepage Enclave and Tenants. National Clinical Cohort Collaborative. URL: https:\/\/covid.cd2h.org\/ [Accessed 21-12-2023]"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Jones M Winger A Wernz C et al. Investigating the impact of temporal labeling of emergency department visits for COVID-19: comparing healthcare disparities analyses using comprehensive, single-site data with national COVID cohort collaborative (N3C) data. Presented at: 2023 Systems and Information Engineering Design Symposium (SIEDS); Apr 27-28, 2023:297-302; Charlottesville, VA. [doi: 10.1109\/SIEDS58326.2023.10137801]","DOI":"10.1109\/SIEDS58326.2023.10137801"},{"issue":"20","key":"R39","doi-asserted-by":"publisher","first-page":"2232","DOI":"10.1200\/JCO.21.01074","article-title":"Outcomes of COVID-19 in patients with cancer: report from the National COVID Cohort Collaborative (N3C)","volume":"39","author":"Sharafeldin","journal-title":"J Clin Oncol"},{"issue":"6","key":"R40","doi-asserted-by":"publisher","first-page":"1125","DOI":"10.1093\/jamia\/ocad057","article-title":"Clinical encounter heterogeneity and methods for resolving in networked EHR data: a study from N3C and RECOVER programs","volume":"30","author":"Leese","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","journal-title":"IEEE Trans Evol Computat"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Patel B Ngana LP Haghighathoseini A Wojtusiak J . Influence of stratified variable encoding on quality of mortality prediction in systolic heart failure. Presented at: 2025 International Conference on Machine Learning and Applications (ICMLA); Dec 3-5, 2025:996-999; Boca Raton, FL. [doi: 10.1109\/ICMLA66185.2025.00150]","DOI":"10.1109\/ICMLA66185.2025.00150"},{"issue":"16","key":"R43","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph18168530","article-title":"Comparison of random forest and gradient boosting machine models for predicting demolition waste based on small datasets and categorical variables","volume":"18","author":"Cha","journal-title":"Int J Environ Res Public Health"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.03.002","article-title":"Logistic regression was as good as machine learning for predicting major chronic diseases","volume":"122","author":"Nusinovici","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"R45","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1161\/CIRCULATIONAHA.114.014508","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group","volume":"131","author":"Collins","journal-title":"Circulation"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2024.104656","article-title":"Causal fairness assessment of treatment allocation with electronic health records","volume":"155","author":"Zhang","journal-title":"J Biomed Inform"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Kecki V Said A . Understanding fairness in recommender systems: a healthcare perspective. Presented at: RecSys \u201924: Proceedings of the 18th ACM Conference on Recommender Systems; Oct 14-18, 2024:1125-1130; Bari, Italy. [doi: 10.1145\/3640457.3691711]","DOI":"10.1145\/3640457.3691711"},{"key":"R48","doi-asserted-by":"crossref","unstructured":"Pfohl S Marafino B Coulet A Rodriguez F Palaniappan L Shah NH . Creating fair models of atherosclerotic cardiovascular disease risk. Presented at: AIES \u201919: Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society; Jan 27-28, 2019:271-278; Honolulu HI. [doi: 10.1145\/3306618.3314278]","DOI":"10.1145\/3306618.3314278"},{"issue":"1","key":"R49","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-026-02768-6","article-title":"Sample size and power analysis for ROC AUC differences in diagnostic tests: a methodological evaluation of the Obuchowski-McClish and Hanley-McNeil methods","volume":"26","author":"Emir","journal-title":"BMC Med Res Methodol"},{"issue":"3","key":"R50","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1093\/jamia\/ocaa196","article-title":"The National COVID Cohort Collaborative (N3C): rationale, design, infrastructure, and deployment","volume":"28","author":"Haendel","journal-title":"J Am Med Inform Assoc"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T14:52:28Z","timestamp":1787237548000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e88406"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,20]]},"references-count":51,"URL":"https:\/\/doi.org\/10.2196\/88406","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,20]]},"article-number":"v10i14e88406"}}