{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T03:36:32Z","timestamp":1790652992761,"version":"4.1.0"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2017,12,23]],"date-time":"2017-12-23T00:00:00Z","timestamp":1513987200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["T32-NS048005"],"award-info":[{"award-number":["T32-NS048005"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["F31-GM119263"],"award-info":[{"award-number":["F31-GM119263"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U54-H6007963"],"award-info":[{"award-number":["U54-H6007963"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01-GM079330"],"award-info":[{"award-number":["R01-GM079330"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,6,1]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>In many modern machine learning applications, the outcome is expensive or time consuming to collect whereas the predictor information is easy to obtain. Semi-supervised (SS) learning aims at utilizing large amounts of \u2018unlabelled\u2019 data along with small amounts of \u2018labelled\u2019 data to improve the efficiency of a classical supervised approach. Though numerous SS learning classification and prediction procedures have been proposed in recent years, no methods currently exist to evaluate the prediction performance of a working regression model. In the context of developing phenotyping algorithms derived from electronic medical records, we present an efficient two-step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the SS setting. In step I, the labelled data are used to obtain a non-parametrically calibrated estimate of the conditional risk function. In step II, SS estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabelled data. We demonstrate that, under mild regularity conditions, the estimators proposed are consistent and asymptotically normal. Importantly, the asymptotic variance of the SS estimators is always smaller than that of the supervised counterparts under correct model specification. We also correct for potential overfitting bias in the SS estimators in finite samples with cross-validation and we develop a perturbation resampling procedure to approximate their distributions. Our proposals are evaluated through extensive simulation studies and illustrated with two real electronic medical record studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis.<\/jats:p>","DOI":"10.1111\/rssb.12264","type":"journal-article","created":{"date-parts":[[2017,12,24]],"date-time":"2017-12-24T10:25:45Z","timestamp":1514111145000},"page":"579-594","source":"Crossref","is-referenced-by-count":32,"title":["Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance"],"prefix":"10.1093","volume":"80","author":[{"given":"Jessica L.","family":"Gronsbell","sequence":"first","affiliation":[{"name":"Harvard University , Boston , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianxi","family":"Cai","sequence":"additional","affiliation":[{"name":"Harvard University , Boston , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2017,12,23]]},"reference":[{"key":"2023091418505117400_rssb12264-bib-0001","doi-asserted-by":"crossref","first-page":"1411","DOI":"10.1097\/MIB.0b013e31828133fd","article-title":"Improving case definition of Crohn's disease and ulcerative colitis in electronic medical records using natural language processing: a novel informatics approach","volume":"19","author":"Ananthakrishnan","year":"2013","journal-title":"Inflamm. Bowel Dis."},{"key":"2023091418505117400_rssb12264-bib-0002","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1093\/jnci\/95.7.511","article-title":"The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer","volume":"95","author":"Baker","year":"2003","journal-title":"J. Natn Cancer Inst."},{"key":"2023091418505117400_rssb12264-bib-0003","first-page":"854","article-title":"Probabilistic modeling for face orientation discrimination learning from labeled and unlabeled data","volume-title":"Advances in Neural Information Processing Systems","author":"Baluja","year":"1998"},{"key":"2023091418505117400_rssb12264-bib-0004","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1023\/B:MACH.0000033120.25363.1e","article-title":"Semi-supervised learning on riemannian manifolds","volume":"56","author":"Belkin","year":"2004","journal-title":"Mach. Learn."},{"key":"2023091418505117400_rssb12264-bib-0005","first-page":"2399","article-title":"Manifold regularization: a geometric framework for learning from labeled and unlabeled examples","volume":"7","author":"Belkin","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"2023091418505117400_rssb12264-bib-0006","first-page":"1071","article-title":"On some global measures of the deviations of density function estimates","volume":"3","author":"Bickel","year":"1973","journal-title":"Ann. Statist."},{"key":"2023091418505117400_rssb12264-bib-0007","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262033589.001.0001","volume-title":"Semi-supervised Learning","author":"Chapelle","year":"2006"},{"key":"2023091418505117400_rssb12264-bib-0008","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1016\/j.cviu.2015.05.007","article-title":"Semi-supervised learning and feature evaluation for rgb-d object recognition","volume":"139","author":"Cheng","year":"2015","journal-title":"Comput. Visn Im. Understand."},{"key":"2023091418505117400_rssb12264-bib-0009","article-title":"Assessing binary classifiers using only positive and unlabeled data","volume-title":"Preprint arXiv:1504.06837","author":"Claesen","year":"2015"},{"key":"2023091418505117400_rssb12264-bib-0010","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1145\/1390156.1390177","article-title":"A unified architecture for natural language processing: deep neural networks with multitask learning","volume-title":"Proc. 25th Int. Conf. Machine Learning","author":"Collobert","year":"2008"},{"key":"2023091418505117400_rssb12264-bib-0011","first-page":"327","article-title":"Unlabeled data can degrade classification performance of generative classifiers","volume-title":"Proc. FLAIRS Conf.","author":"Cozman","year":"2002"},{"key":"2023091418505117400_rssb12264-bib-0012","first-page":"99","article-title":"Semi-supervised learning of mixture models","volume-title":"Proc. Int. Conf. Machine Learning","author":"Cozman","year":"2003"},{"key":"2023091418505117400_rssb12264-bib-0013","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1080\/01621459.1986.10478291","article-title":"How biased is the apparent error rate of a prediction rule","volume":"81","author":"Efron","year":"1986","journal-title":"J. Am. Statist. Ass"},{"key":"2023091418505117400_rssb12264-bib-0014","doi-asserted-by":"crossref","first-page":"1348","DOI":"10.1198\/016214501753382273","article-title":"Variable selection via nonconcave penalized likelihood and its oracle properties","volume":"96","author":"Fan","year":"2001","journal-title":"J. Am. Statist. Ass."},{"key":"2023091418505117400_rssb12264-bib-0015","first-page":"529","article-title":"Semi-supervised learning by entropy minimization","volume-title":"Advances in Neural Information Processing Systems","author":"Grandvalet","year":"2004"},{"key":"2023091418505117400_rssb12264-bib-0016","first-page":"487","article-title":"Exploiting generative models in discriminative classifiers","volume-title":"Advances in Neural Information Processing Systems","author":"Jaakkola","year":"1999"},{"key":"2023091418505117400_rssb12264-bib-0017","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1093\/biomet\/88.2.381","article-title":"A simple resampling method by perturbing the minimand","volume":"88","author":"Jin","year":"2001","journal-title":"Biometrika"},{"key":"2023091418505117400_rssb12264-bib-0018","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1038\/nrg2999","article-title":"Using electronic health records to drive discovery in disease genomics","volume":"12","author":"Kohane","year":"2011","journal-title":"Nat. Rev. Genet."},{"key":"2023091418505117400_rssb12264-bib-0019","first-page":"307","article-title":"Challenges in statistical machine learning","volume":"16","author":"Lafferty","year":"2006","journal-title":"Statist. Sin."},{"key":"2023091418505117400_rssb12264-bib-0020","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1002\/acr.20184","article-title":"Electronic medical records for discovery research in rheumatoid arthritis","volume":"62","author":"Liao","year":"2010","journal-title":"Arth. Care Res."},{"key":"2023091418505117400_rssb12264-bib-0021","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.h1885","article-title":"Development of phenotype algorithms using electronic medical records and incorporating natural language processing","volume":"350","author":"Liao","year":"2015","journal-title":"Br. Med. J."},{"key":"2023091418505117400_rssb12264-bib-0022","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1002\/art.37801","article-title":"Autoantibodies, autoimmune risk alleles and clinical associations in rheumatoid arthritis cases and non-ra controls in the electronic medical records","volume":"65","author":"Liao","year":"2012","journal-title":"Arth. Rheum."},{"key":"2023091418505117400_rssb12264-bib-0023","first-page":"856","article-title":"Graph-based semi-supervised learning algorithm for web page classification","volume-title":"Proc. 6th Int. Conf. Intelligent Systems Design and Applications","author":"Liu","year":"2006"},{"key":"2023091418505117400_rssb12264-bib-0024","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1111\/j.0006-341X.2002.00657.x","article-title":"Combining several screening tests: optimality of the risk score","volume":"58","author":"McIntosh","year":"2002","journal-title":"Biometrics"},{"key":"2023091418505117400_rssb12264-bib-0025","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1198\/jasa.2011.tm10382","article-title":"A perturbation method for inference on regularized regression estimates","volume":"106","author":"Minnier","year":"2011","journal-title":"J. Am. Statist. Ass."},{"key":"2023091418505117400_rssb12264-bib-0026","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1101\/gr.094615.109","article-title":"Instrumenting the health care enterprise for discovery research in the genomic era","volume":"19","author":"Murphy","year":"2009","journal-title":"Genome Res."},{"key":"2023091418505117400_rssb12264-bib-0027","doi-asserted-by":"crossref","first-page":"33","DOI":"10.7551\/mitpress\/6173.003.0006","article-title":"Semi-supervised text classification using EM","volume-title":"Semi-supervised Learning","author":"Nigam","year":"2006"},{"key":"2023091418505117400_rssb12264-bib-0028","first-page":"1229","article-title":"Manifold regularization and semi-supervised learning: some theoretical analyses","volume":"14","author":"Niyogi","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"2023091418505117400_rssb12264-bib-0029","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1080\/01621459.1978.10480106","article-title":"Normal discrimination with unclassified observations","volume":"73","author":"O\u2019Neill","year":"1978","journal-title":"J. Am. Statist. Ass."},{"key":"2023091418505117400_rssb12264-bib-0030","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198509844.001.0001","volume-title":"The Statistical Evaluation of Medical Tests for Classification and Prediction","author":"Pepe","year":"2003"},{"key":"2023091418505117400_rssb12264-bib-0031","first-page":"29","article-title":"Semi-supervised self-training of object detection models","volume-title":"Proc. 7th Wrkshp. Application of Computer Vision","author":"Rosenberg","year":"2005"},{"key":"2023091418505117400_rssb12264-bib-0032","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1093\/biomet\/63.3.581","article-title":"Inference and missing data","volume":"63","author":"Rubin","year":"1976","journal-title":"Biometrika"},{"key":"2023091418505117400_rssb12264-bib-0033","article-title":"Learning with labeled and unlabeled data","volume-title":"Technical Report","author":"Seeger","year":"2000"},{"key":"2023091418505117400_rssb12264-bib-0034","first-page":"1513","article-title":"Unlabeled data: now it helps, now it doesn\u2019t","volume-title":"Advances in Neural Information Processing Systems","author":"Singh","year":"2009"},{"key":"2023091418505117400_rssb12264-bib-0035","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1007\/s00439-014-1466-9","article-title":"Improving the power of genetic association tests with imperfect phenotype derived from electronic medical records","volume":"133","author":"Sinnott","year":"2014","journal-title":"Hum. Genet."},{"key":"2023091418505117400_rssb12264-bib-0036","first-page":"1","article-title":"Semi-supervised learning and domain adaptation in natural language processing","volume-title":"In Synthesis Lectures on Human Language Technologies","author":"S\u00f8gaard","year":"2013"},{"key":"2023091418505117400_rssb12264-bib-0037","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1093\/biomet\/asm036","article-title":"Model evaluation based on the sampling distribution of estimated absolute prediction error","volume":"94","author":"Tian","year":"2007","journal-title":"Biometrika"},{"key":"2023091418505117400_rssb12264-bib-0038","first-page":"242","article-title":"Web page classification based on semi-supervised na\u00efve bayesian em algorithm","volume-title":"Proc. 3rd Int. Conf. Communication Software and Networks","author":"Wang","year":"2011"},{"key":"2023091418505117400_rssb12264-bib-0039","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.1198\/016214507000000509","article-title":"Unified lasso estimation by least squares approximation","volume":"102","author":"Wang","year":"2007","journal-title":"J. Am. Statist. Ass."},{"key":"2023091418505117400_rssb12264-bib-0040","first-page":"801","article-title":"Statistical analysis of semi-supervised regression","volume-title":"Advances in Neural Information Processing Systems","author":"Wasserman","year":"2008"},{"key":"2023091418505117400_rssb12264-bib-0041","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1038\/clpt.2010.260","article-title":"The emerging role of electronic medical records in pharmacogenomics","volume":"89","author":"Wilke","year":"2011","journal-title":"Clin. Pharmcol. Therp."},{"key":"2023091418505117400_rssb12264-bib-0042","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0078927","article-title":"Modeling disease severity in multiple sclerosis using electronic health records","volume":"8","author":"Xia","year":"2013","journal-title":"PLOS ONE"},{"key":"2023091418505117400_rssb12264-bib-0043","first-page":"1191","article-title":"The value of unlabeled data for classification problems","volume-title":"Proc. 17th Int. Conf. Machine Learning","author":"Zhang","year":"2000"},{"key":"2023091418505117400_rssb12264-bib-0044","article-title":"Semi-supervised learning literature survey","volume-title":"Technical Report","author":"Zhu","year":"2006"},{"key":"2023091418505117400_rssb12264-bib-0045","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1198\/016214506000000735","article-title":"The adaptive lasso and its oracle properties","volume":"101","author":"Zou","year":"2006","journal-title":"J. Am. Statist. Ass."},{"key":"2023091418505117400_rssb12264-bib-0046","first-page":"1509","article-title":"One-step sparse estimates in nonconcave penalized likelihood models","volume":"36","author":"Zou","year":"2008","journal-title":"Ann. Statist."}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Frssb.12264","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/80\/3\/579\/51555543\/jrsssb_80_3_579.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/80\/3\/579\/51555543\/jrsssb_80_3_579.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,30]],"date-time":"2024-06-30T02:30:32Z","timestamp":1719714632000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/80\/3\/579\/7048446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,23]]},"references-count":46,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2017,12,23]]},"published-print":{"date-parts":[[2018,6,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12264","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,6]]},"published":{"date-parts":[[2017,12,23]]}}}