{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:59:45Z","timestamp":1782845985781,"version":"3.54.5"},"reference-count":52,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Developing computable phenotypes (CP) based on electronic health records (EHR) data requires \u201cgold-standard\u201d labels for the outcome of interest. To generate these labels, clinicians typically chart-review a subset of patient charts. Charts to be reviewed are most often randomly sampled from the larger set of patients of interest. However, random sampling may fail to capture the diversity of the patient population, particularly if smaller subpopulations exist among those with the condition of interest. This can lead to poorly performing and biased CPs.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to propose an unsupervised sampling approach designed to better capture a diverse patient cohort and improve the information coverage of chart review samples.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Our coverage sampling method starts by clustering by the patient population of interest. We then perform a stratified sampling from each cluster to ensure even representation for the chart review sample. We introduce a novel metric, nearest neighbor distance, to evaluate the coverage of the generated sample. To evaluate our method, we first conducted a simulation study to model and compare the performance of random versus our proposed coverage sampling. We varied the size and number of subpopulations within the larger cohort. Finally, we apply our approach to a real-world data set to develop a CP for hospitalization due to COVID-19. We evaluate the different sampling strategies based on the information coverage as well as the performance of the learned CP, using the area under the receiver operator characteristic curve.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Our simulation studies show that the unsupervised coverage sampling approach provides broader coverage of patient populations compared to random sampling. When there are no underlying subpopulations, both random and coverage perform equally well for CP development. When there are subgroups, coverage sampling achieves area under the receiver operating characteristic curve gains of approximately 0.03\u20100.05 over random sampling. In the real-world application, the approach also outperformed random sampling, generating both a more representative sample and an area under the receiver operating characteristic curve improvement of 0.02 (95% CI \u22120.08 to 0.04).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The proposed coverage sampling method is an easy-to-implement approach that produces a chart review sample that is more representative of the source population. This allows one to learn a CP that has better performance both for subpopulations and the overall cohort. Studies that aim to develop CPs should consider alternative strategies other than randomly sampling patient charts.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72068","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:55:07Z","timestamp":1760036107000},"page":"e72068-e72068","source":"Crossref","is-referenced-by-count":0,"title":["Unsupervised Coverage Sampling to Enhance Clinical Chart Review Coverage for Computable Phenotype Development: Simulation and Empirical Study"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1472-593X","authenticated-orcid":false,"given":"Zigui","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5079-9920","authenticated-orcid":false,"given":"Jillian H","family":"Hurst","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7056-9559","authenticated-orcid":false,"given":"Chuan","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5261-3632","authenticated-orcid":false,"given":"Benjamin Alan","family":"Goldstein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,11,27]]},"reference":[{"issue":"8","key":"R1","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1093\/jamia\/ocaa125","article-title":"Characterizing outpatient problem list completeness and duplications in the electronic health record","volume":"27","author":"Wang","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","first-page":"909","DOI":"10.1093\/jamia\/ocab285","article-title":"Indication alerts to improve problem list documentation","volume":"29","author":"Grauer","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1515\/jpem.2007.20.3.359","article-title":"The diagnosis of children with central diabetes insipidus","volume":"20","author":"Ghirardello","journal-title":"J Pediatr Endocrinol Metab"},{"issue":"6","key":"R4","doi-asserted-by":"publisher","first-page":"1430","DOI":"10.1016\/j.jaci.2021.10.001","article-title":"Advances in asthma: New understandings of asthma\u2019s natural history, risk factors, underlying mechanisms, and clinical management","volume":"148","author":"Miller","journal-title":"J Allergy Clin Immunol"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.3389\/fgene.2020.00556","article-title":"Impact of diverse data sources on computational phenotyping","volume":"11","author":"Wang","journal-title":"Front Genet"},{"issue":"3","key":"R6","first-page":"126","volume":"15","author":"Gearing","journal-title":"J Can Acad Child Adolesc Psychiatry"},{"issue":"5","key":"R7","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1016\/j.amj.2007.06.007","article-title":"Performing chart review studies","volume":"26","author":"Panacek","journal-title":"Air Med J"},{"issue":"11","key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/29241","article-title":"A semiautomated chart review for assessing the development of radiation pneumonitis using natural language processing: diagnostic accuracy and feasibility study","volume":"9","author":"McKenzie","journal-title":"JMIR Med Inform"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2014.07.001","article-title":"Limestone: high-throughput candidate phenotype generation via tensor factorization","volume":"52","author":"Ho","journal-title":"J Biomed Inform"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","first-page":"1785","DOI":"10.1093\/jamia\/ocae121","article-title":"A general framework for developing computable clinical phenotype algorithms","volume":"31","author":"Carrell","journal-title":"J Am Med Inform Assoc"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3352\/jeehp.2013.10.12","article-title":"The retrospective chart review: important methodological considerations","volume":"10","author":"Vassar","journal-title":"J Educ Eval Health Prof"},{"issue":"1","key":"R12","volume":"5","author":"Elfil","journal-title":"Emerg (Tehran)"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1207\/S15327906Mb340201","article-title":"A method for generating simulated plasmodes and artificial test clusters with user-defined shape, size, and orientation","volume":"34","author":"Waller","journal-title":"Multivariate Behav Res"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/46267","article-title":"Comparing natural language processing and structured medical data to develop a computable phenotype for patients hospitalized due to COVID-19: retrospective analysis","volume":"11","author":"Chang","journal-title":"JMIR Med Inform"},{"issue":"5","key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/37931","article-title":"Distinguishing admissions specifically for COVID-19 from incidental SARS-CoV-2 admissions: National Retrospective Electronic Health Record Study","volume":"24","author":"Klann","journal-title":"J Med Internet Res"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/cts.2020.499","article-title":"Development of an electronic health records datamart to support clinical and population health research","volume":"5","author":"Hurst","journal-title":"J Clin Trans Sci"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1017\/pan.2020.4","article-title":"Active learning approaches for labeling text: review and assessment of the performance of active learning approaches","volume":"28","author":"Miller","journal-title":"Polit Anal"},{"key":"R18","unstructured":"Cohn DA Ghahramani Z Jordan MI . Active learning with statistical models [abstract]. Presented at: Advances in Neural Information Processing Systems 7 (NIPS 1994); Nov 28 to Dec 1, 1994; Denver, Colorado. 1996.URL: https:\/\/papers.nips.cc\/paper_files\/paper\/1994\/hash\/7f975a56c761db6506eca0b37ce6ec87-Abstract.html [Accessed 11-11-2025]"},{"issue":"10","key":"R19","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1109\/TPAMI.2014.2307881","article-title":"Active learning by querying informative and representative examples","volume":"36","author":"Huang","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Sinha S Ebrahimi S Darrell T . Variational adversarial active learning. Presented at: 2019 IEEE\/CVF International Conference on Computer Vision (ICCV); Seoul, South Korea. Mar 31, 2019.[doi: 10.1109\/ICCV.2019.00607]","DOI":"10.1109\/ICCV.2019.00607"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Li X Guo Y . Adaptive active learning for image classification. Presented at: IEEE Conference on Computer Vision and Pattern Recognition; Jun 23-28, 2013:859-866; Portland, OR, USA. 2013.[doi: 10.1109\/CVPR.2013.116]","DOI":"10.1109\/CVPR.2013.116"},{"key":"R22","unstructured":"Wang Z . phenotype-sampling. URL: https:\/\/github.com\/justinwang97\/phenotype-sampling [Accessed 19-07-2025]"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/16042","article-title":"Clinical Annotation Research Kit (CLARK): computable phenotyping using machine learning","volume":"8","author":"Pfaff","journal-title":"JMIR Med Inform"},{"issue":"6","key":"R24","doi-asserted-by":"publisher","DOI":"10.1002\/wics.1549","article-title":"Challenges and opportunities beyond structured data in analysis of electronic health records","volume":"13","author":"Tayefi","journal-title":"WIREs Computational Stats"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-018-0657-9","article-title":"Methods of sample size calculation in descriptive retrospective burden of illness studies","volume":"19","author":"Johnston","journal-title":"BMC Med Res Methodol"},{"issue":"2","key":"R26","doi-asserted-by":"publisher","DOI":"10.1093\/intqhc\/mzae037","article-title":"A scoping review of the methodological approaches used in retrospective chart reviews to validate adverse event rates in administrative data","volume":"36","author":"Connolly","journal-title":"Int J Qual Health Care"},{"issue":"e2","key":"R27","doi-asserted-by":"publisher","first-page":"e319","DOI":"10.1136\/amiajnl-2013-001952","article-title":"A comparison of phenotype definitions for diabetes mellitus","volume":"20","author":"Richesson","journal-title":"J Am Med Inform Assoc"},{"issue":"e1","key":"R28","doi-asserted-by":"publisher","first-page":"e121","DOI":"10.1093\/jamia\/ocw123","article-title":"Assessing electronic health record phenotypes against gold-standard diagnostic criteria for diabetes mellitus","volume":"24","author":"Spratt","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"391","DOI":"10.2307\/1933159","article-title":"Distribution of distance to Nth neighbour in a population of randomly distributed individuals","volume":"37","author":"Thompson","journal-title":"Ecology"},{"key":"R30","volume":"5","author":"Peet","journal-title":"Annu Rev Ecol Syst"},{"issue":"4148","key":"R31","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1038\/163688a0","article-title":"Measurement of diversity","volume":"163","author":"Simpson","journal-title":"Nature New Biol"},{"issue":"3","key":"R32","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","journal-title":"Bell System Technical Journal"},{"key":"R33","unstructured":"Settles B . Active learning literature survey. Minds@UW; 2009. URL: https:\/\/minds.wisconsin.edu\/handle\/1793\/60660 [Accessed 24-11-2025]"},{"issue":"2","key":"R34","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1023\/A:1007330508534","article-title":"Selective sampling using the query by committee algorithm","volume":"28","author":"Freund","journal-title":"Mach Learn"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.5555\/645531.655845","article-title":"Active semi-supervised learning = robust multi-view learning","volume":"2","author":"Muslea","journal-title":"ICML \u201902: Proceedings of the Nineteenth International Conference on Machine Learning"},{"key":"R36","unstructured":"McCallum A Nigam K . Employing EM and pool-based active learning for text classification. Presented at: Proceedings of the 15th International Conference on Machine Learning; Jul 24-27, 1998:350-358; Madison, WI, USA. 1998.[doi: 10.5555\/645527.757765]"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Jain P Kapoor A . Active learning for large multi-class problems. Presented at: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops); Jun 20-25, 2009:762-769; Miami, FL, USA. 2009.[doi: 10.1109\/CVPR.2009.5206651]","DOI":"10.1109\/CVPR.2009.5206651"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Kapoor A Grauman K Urtasun R Darrell T . Active learning with gaussian processes for object categorization. Presented at: 2007 IEEE 11th International Conference on Computer Vision. 1-8; 2007.[doi: 10.1109\/ICCV.2007.4408844]","DOI":"10.1109\/ICCV.2007.4408844"},{"issue":"4","key":"R39","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1162\/neco.1992.4.4.590","article-title":"Information-based objective functions for active data selection","volume":"4","author":"MacKay","journal-title":"Neural Comput"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760185243","article-title":"Support vector machine active learning with applications to text classification","volume":"2","author":"Tong","journal-title":"J Mach Learn Res"},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Suresh H Guttag J . A framework for understanding sources of harm throughout the machine learning life cycle. Presented at: 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO \u201921); Oct 5-9, 2021. Oct 5, 2021.[doi: 10.1145\/3465416.3483305]","DOI":"10.1145\/3465416.3483305"},{"issue":"9","key":"R42","doi-asserted-by":"publisher","first-page":"1176","DOI":"10.1001\/archneurol.2012.314","article-title":"Epidemiology of neuromyelitis optica in the United States: a multicenter analysis","volume":"69","author":"Mealy","journal-title":"Arch Neurol"},{"issue":"6","key":"R43","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1016\/j.chest.2017.12.018","article-title":"The challenge of rare diseases","volume":"153","author":"Stoller","journal-title":"Chest"},{"key":"R44","unstructured":"Buolamwini J Gebru T . Gender shades: intersectional accuracy disparities in commercial gender classification. In: Friedler SA Wilson C , editors. Presented at: Proceedings of the 1st Conference on Fairness, Accountability and Transparency; 77-91; New York, NY, USA. May 2018.URL: https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html [Accessed 19-11-2025]"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Kennedy G Dras M Gallego B . Augmentation of electronic medical record data for deep learning. In: In Studies in Health Technology and Informatics. IOS Press BV; 2022:582-586. [doi: 10.3233\/SHTI220144]","DOI":"10.3233\/SHTI220144"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","journal-title":"J Big Data"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla","journal-title":"jair"},{"key":"R48","doi-asserted-by":"crossref","unstructured":"Kim YT Kim DK Kim H Kim DJ . A comparison of oversampling methods for constructing a prognostic model in the patient with heart failure. Presented at: 2020 International Conference on Information and Communication Technology Convergence (ICTC); Oct 21-23, 2020; Jeju Island, South Korea. [doi: 10.1109\/ICTC49870.2020.9289522]","DOI":"10.1109\/ICTC49870.2020.9289522"},{"key":"R49","unstructured":"Hino H . Active learning: problem settings and recent developments. arXiv. Preprint posted online on Dec 8, 2020. [doi: 10.48550\/arXiv.2012.04225]"},{"issue":"8","key":"R50","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1016\/j.patrec.2009.09.011","article-title":"Data clustering: 50 years beyond K-means","volume":"31","author":"Jain","journal-title":"Pattern Recognit Lett"},{"key":"R51","doi-asserted-by":"crossref","unstructured":"Hershey JR Olsen PA . Approximating the kullback leibler divergence between Gaussian mixture models. Presented at: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP \u201907; IV-317; Honolulu, HI, USA. 2007.[doi: 10.1109\/ICASSP.2007.366913]","DOI":"10.1109\/ICASSP.2007.366913"},{"issue":"2","key":"R52","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1093\/oxfordjournals.pan.a004868","article-title":"Logistic regression in rare events data","volume":"9","author":"King","journal-title":"Polit anal"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e72068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:10Z","timestamp":1782842530000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e72068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,27]]},"references-count":52,"URL":"https:\/\/doi.org\/10.2196\/72068","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,27]]},"article-number":"v13i8e72068"}}