{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T11:56:16Z","timestamp":1767182176513,"version":"3.37.3"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T00:00:00Z","timestamp":1703116800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN-2021-03734"],"award-info":[{"award-number":["RGPIN-2021-03734"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"name":"University of Toronto Connaught New Researcher"},{"name":"University of Toronto Seed Funding for Methodologists"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (eg, sensitivity, specificity).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC with synthetic, semi-synthetic, and EHR data from Mass General Brigham (MGB).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>ssROC produced ROC parameter estimates with minimal bias and significantly lower variance than supROC in the simulated and semi-synthetic data. For the 5 PAs from MGB, the estimates from ssROC are 30% to 60% less variable than supROC on average.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>ssROC enables precise evaluation of PA performance without demanding large volumes of labeled data. ssROC is also easily implementable in open-source R software.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad226","type":"journal-article","created":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T22:15:52Z","timestamp":1703196952000},"page":"640-650","source":"Crossref","is-referenced-by-count":4,"title":["Semi-supervised ROC analysis for reliable and streamlined evaluation of phenotyping algorithms"],"prefix":"10.1093","volume":"31","author":[{"given":"Jianhui","family":"Gao","sequence":"first","affiliation":[{"name":"Department of Statistical Sciences, University of Toronto , Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Clara-Lea","family":"Bonzel","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chuan","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke University , Durham, NC, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul","family":"Varghese","sequence":"additional","affiliation":[{"name":"Health Informatics, Verily Life Sciences , Cambridge, MA, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karim","family":"Zakir","sequence":"additional","affiliation":[{"name":"Department of Statistical Sciences, University of Toronto , Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jessica","family":"Gronsbell","sequence":"additional","affiliation":[{"name":"Department of Statistical Sciences, University of Toronto , Toronto, ON, Canada"},{"name":"Department of Family and Community Medicine, University of Toronto , Toronto, ON, Canada"},{"name":"Department of Computer Science, University of Toronto , Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2023,12,21]]},"reference":[{"volume-title":"Clinical Data as the Basic Staple of Health Learning: Creating and Protecting a Public Good: Workshop Summary","year":"2011","author":"McGinnis","key":"2024021710233261000_ocad226-B1"},{"issue":"5","key":"2024021710233261000_ocad226-B2","first-page":"e16","article-title":"Electronic health records and adverse drug events after patient transfer","volume":"19","author":"Boockvar","year":"2010","journal-title":"Qual Saf Health Care"},{"issue":"1","key":"2024021710233261000_ocad226-B3","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ajhg.2010.12.007","article-title":"Genetic basis of autoantibody positive and negative rheumatoid arthritis risk in a multi-ethnic cohort derived from electronic health records","volume":"88","author":"Kurreeman","year":"2011","journal-title":"Am J Hum Genet"},{"issue":"3","key":"2024021710233261000_ocad226-B4","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1002\/art.37801","article-title":"Associations of autoantibodies, autoimmune risk alleles, and clinical diagnoses from the electronic medical records in rheumatoid arthritis cases and non-rheumatoid arthritis controls","volume":"65","author":"Liao","year":"2013","journal-title":"Arthritis Rheum"},{"issue":"1","key":"2024021710233261000_ocad226-B5","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1038\/s41398-018-0133-7","article-title":"Genetic validation of bipolar disorder identified by automated phenotyping using electronic health records","volume":"8","author":"Chen","year":"2018","journal-title":"Transl Psychiatry"},{"issue":"8","key":"2024021710233261000_ocad226-B6","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1038\/s41576-020-0224-1","article-title":"Electronic health records and polygenic risk scores for predicting disease risk","volume":"21","author":"Li","year":"2020","journal-title":"Nat Rev Genet"},{"key":"2024021710233261000_ocad226-B7","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1038\/s41746-020-00308-0","article-title":"International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium","volume":"3","author":"Brat","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2024021710233261000_ocad226-B8","first-page":"1","article-title":"Using phecodes for research with the electronic health record: from PheWAS to PheRS","volume":"4","author":"Bastarache","year":"2021"},{"key":"2024021710233261000_ocad226-B9","first-page":"369","article-title":"Unraveling COVID-19: a large-scale characterization of 4.5 million COVID-19 cases using CHARYBDIS","volume":"14","author":"Prieto-Alhambra","year":"2021","journal-title":"Res Square"},{"issue":"7","key":"2024021710233261000_ocad226-B10","doi-asserted-by":"crossref","first-page":"1447","DOI":"10.1038\/s41591-022-01895-z","article-title":"Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing","volume":"28","author":"Henry","year":"2022","journal-title":"Nature Med"},{"issue":"2","key":"2024021710233261000_ocad226-B11","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1136\/amiajnl-2013-001935","article-title":"A review of approaches to identifying patient phenotype cohorts using electronic health records","volume":"21","author":"Shivade","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B12","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: from rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu Rev Biomed Data Sci"},{"issue":"11","key":"2024021710233261000_ocad226-B13","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.3390\/electronics8111235","article-title":"A review of automatic phenotyping approaches using electronic health records","volume":"8","author":"Alzoubi","year":"2019","journal-title":"Electronics"},{"issue":"2","key":"2024021710233261000_ocad226-B14","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1093\/jamia\/ocac216","article-title":"Machine learning approaches for electronic health records phenotyping: a methodical review","volume":"30","author":"Yang","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"2024021710233261000_ocad226-B15","doi-asserted-by":"crossref","first-page":"3426","DOI":"10.1038\/s41596-019-0227-6","article-title":"High-throughput phenotyping with electronic medical record data using a common semi-supervised approach (PheCAP)","volume":"14","author":"Zhang","year":"2019","journal-title":"Nat Protoc"},{"issue":"9","key":"2024021710233261000_ocad226-B16","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1101\/gr.094615.109","article-title":"Instrumenting the health care enterprise for discovery research in the genomic era","volume":"19","author":"Murphy","year":"2009","journal-title":"Genome Res"},{"key":"2024021710233261000_ocad226-B17","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1186\/s12958-015-0115-z","article-title":"Identification of subjects with polycystic ovary syndrome using electronic health records","volume":"13","author":"Castro","year":"2015","journal-title":"Reprod Biol Endocrinol"},{"issue":"1","key":"2024021710233261000_ocad226-B18","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1093\/jamia\/ocw071","article-title":"Evaluating electronic health record data sources and algorithmic approaches to identify hypertensive individuals","volume":"24","author":"Teixeira","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B19","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.jpeds.2017.05.037","article-title":"A computable phenotype improves cohort ascertainment in a pediatric pulmonary hypertension registry","volume":"188","author":"Geva","year":"2017","journal-title":"J Pediatr"},{"issue":"4","key":"2024021710233261000_ocad226-B20","article-title":"Using biomedical text as data and representation learning for identifying patients with an osteoarthritis phenotype in the electronic medical record","volume":"3","author":"Meaney","year":"2018","journal-title":"Int J Popul Data Sci"},{"issue":"2","key":"2024021710233261000_ocad226-B21","doi-asserted-by":"crossref","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","article-title":"Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives","volume":"13","author":"Gehrmann","year":"2018","journal-title":"PLoS One"},{"issue":"11","key":"2024021710233261000_ocad226-B22","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1093\/jamia\/ocz066","article-title":"High-throughput multimodal automated phenotyping (MAP) with application to PheWAS","volume":"26","author":"Liao","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"2024021710233261000_ocad226-B23","doi-asserted-by":"crossref","first-page":"e0236400","DOI":"10.1371\/journal.pone.0236400","article-title":"Deep neural network models for identifying incident dementia using claims and EHR datasets","volume":"15","author":"Nori","year":"2020","journal-title":"PLoS One"},{"issue":"10","key":"2024021710233261000_ocad226-B24","doi-asserted-by":"crossref","first-page":"2116","DOI":"10.1093\/jamia\/ocab116","article-title":"Automated detection of substance use information from electronic health records for a pediatric population","volume":"28","author":"Ni","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B25","doi-asserted-by":"crossref","first-page":"103258","DOI":"10.1016\/j.jbi.2019.103258","article-title":"PheValuator: development and evaluation of a phenotype algorithm evaluator","volume":"97","author":"Swerdel","year":"2019","journal-title":"J Biomed Inform"},{"issue":"6","key":"2024021710233261000_ocad226-B26","doi-asserted-by":"crossref","first-page":"e3633","DOI":"10.1097\/GOX.0000000000003633","article-title":"ChartSweep: a HIPAA-compliant tool to automate chart review for plastic surgery research","volume":"9","author":"Chartier","year":"2021","journal-title":"Plast Reconstr Surg Global Open"},{"issue":"5","key":"2024021710233261000_ocad226-B27","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1093\/jamia\/ocv034","article-title":"Toward high-throughput phenotyping: unbiased automated feature extraction and selection from knowledge sources","volume":"22","author":"Yu","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"e1","key":"2024021710233261000_ocad226-B28","doi-asserted-by":"crossref","first-page":"e143","DOI":"10.1093\/jamia\/ocw135","article-title":"Surrogate-assisted feature extraction for high-throughput phenotyping","volume":"24","author":"Yu","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B29","doi-asserted-by":"crossref","first-page":"104175","DOI":"10.1016\/j.jbi.2022.104175","article-title":"Weakly semi-supervised phenotyping using electronic health records","volume":"134","author":"Nogues","year":"2022","journal-title":"J Biomed Inform"},{"issue":"6","key":"2024021710233261000_ocad226-B30","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1016\/j.jbi.2010.09.009","article-title":"An automated technique for identifying associations between medications, laboratory results and problems","volume":"43","author":"Wright","year":"2010","journal-title":"J Biomed Inform"},{"issue":"6","key":"2024021710233261000_ocad226-B31","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1136\/amiajnl-2011-000121","article-title":"A method and knowledge base for automated inference of patient problems from structured data in an electronic medical record","volume":"18","author":"Wright","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024021710233261000_ocad226-B32","doi-asserted-by":"crossref","first-page":"1166","DOI":"10.1093\/jamia\/ocw028","article-title":"Learning statistical models of phenotypes using noisy labeled training data","volume":"23","author":"Agarwal","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B33","first-page":"48","article-title":"Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network","volume":"2017","author":"Banda","year":"2017","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"issue":"3","key":"2024021710233261000_ocad226-B34","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1093\/jamia\/ocx137","article-title":"PIE: a prior knowledge guided integrated likelihood estimation method for bias reduction in association studies using electronic health records data","volume":"25","author":"Huang","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2024021710233261000_ocad226-B35","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1093\/jamia\/ocz180","article-title":"An augmented estimation procedure for EHR-based association studies accounting for differential misclassification","volume":"27","author":"Tong","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024021710233261000_ocad226-B36","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1093\/jamia\/ocab222","article-title":"A cost-effective chart review sampling design to account for phenotyping error in electronic health records (EHR) data","volume":"29","author":"Yin","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B37","doi-asserted-by":"crossref","first-page":"104177","DOI":"10.1016\/j.jbi.2022.104177","article-title":"PheValuator 2.0: methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation","volume":"135","author":"Swerdel","year":"2022","journal-title":"J Biomed Inform"},{"issue":"3","key":"2024021710233261000_ocad226-B38","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1111\/rssb.12264","article-title":"Semi-supervised approaches to efficient evaluation of model prediction performance","volume":"80","author":"Gronsbell","year":"2018","journal-title":"J R Stat Soc B"},{"issue":"4","key":"2024021710233261000_ocad226-B39","first-page":"1353","article-title":"Efficient evaluation of prediction rules in semi-supervised settings under stratified sampling","volume":"84","author":"Gronsbell","year":"2022"},{"issue":"1","key":"2024021710233261000_ocad226-B40","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster","year":"2019","journal-title":"BMC Med"},{"issue":"4","key":"2024021710233261000_ocad226-B41","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1093\/jamia\/ocz228","article-title":"A tutorial on calibration measurements and calibration models for clinical prediction models","volume":"27","author":"Huang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2024021710233261000_ocad226-B42","first-page":"48","article-title":"Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network","volume":"2017","author":"Banda","year":"2017","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"issue":"1","key":"2024021710233261000_ocad226-B43","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1093\/jamia\/ocx111","article-title":"Enabling phenotypic big data with PheNorm","volume":"25","author":"Yu","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024021710233261000_ocad226-B44","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1111\/biom.12987","article-title":"Automated feature selection of predictors in electronic medical records data","volume":"75","author":"Gronsbell","year":"2019","journal-title":"Biometrics"},{"key":"2024021710233261000_ocad226-B45","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198509844.001.0001","volume-title":"The Statistical Evaluation of Medical Tests for Classification and Prediction","author":"Pepe","year":"2003"},{"issue":"1","key":"2024021710233261000_ocad226-B46","first-page":"29","article-title":"The meaning and use of the area under a receiver operating characteristic (ROC) curve","volume":"143","author":"Hanley","year":"1982"},{"issue":"496","key":"2024021710233261000_ocad226-B47","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1198\/jasa.2011.tm10382","article-title":"A perturbation method for inference on regularized regression estimates","volume":"106","author":"Minnier","year":"2011","journal-title":"J Am Stat Assoc"},{"key":"2024021710233261000_ocad226-B48","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"12","key":"2024021710233261000_ocad226-B49","doi-asserted-by":"crossref","first-page":"1102","DOI":"10.1038\/nbt.2749","article-title":"Systematic comparison of phenome-wide association study of electronic medical record data and genome-wide association study data","volume":"31","author":"Denny","year":"2013","journal-title":"Nat Biotechnol"},{"issue":"414","key":"2024021710233261000_ocad226-B50","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1080\/01621459.1991.10475041","article-title":"Transformations in density estimation","volume":"86","author":"Wand","year":"1991","journal-title":"J Am Stat Assoc"},{"key":"2024021710233261000_ocad226-B51","doi-asserted-by":"crossref","DOI":"10.1201\/9781315140919","volume-title":"Density Estimation for Statistics and Data Analysis","author":"Silverman","year":"2018"},{"issue":"4","key":"2024021710233261000_ocad226-B52","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1093\/biostatistics\/kxw016","article-title":"Inference for survival prediction under the regularized Cox model","volume":"17","author":"Sinnott","year":"2016","journal-title":"Biostatistics"},{"volume-title":"Categorical Data Analysis","year":"2012","author":"Agresti","key":"2024021710233261000_ocad226-B53"},{"issue":"2","key":"2024021710233261000_ocad226-B54","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1093\/biostatistics\/kxaa028","article-title":"Surrogate-guided sampling designs for classification of rare outcomes from electronic medical records data","volume":"23","author":"Tan","year":"2022","journal-title":"Biostatistics"},{"issue":"3","key":"2024021710233261000_ocad226-B55","first-page":"2066","article-title":"Double robust semi-supervised inference for the mean: selection bias under MAR labeling with decaying overlap","volume":"12","author":"Zhang","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/3\/640\/56691638\/ocad226.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/3\/640\/56691638\/ocad226.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,17]],"date-time":"2024-02-17T10:24:49Z","timestamp":1708165489000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/3\/640\/7486839"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,21]]},"references-count":55,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,12,21]]},"published-print":{"date-parts":[[2024,2,16]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad226","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2024,3,1]]},"published":{"date-parts":[[2023,12,21]]}}}