{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T03:36:37Z","timestamp":1790652997137,"version":"4.1.0"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In many contemporary applications, large amounts of unlabelled data are readily available while labelled examples are limited. There has been substantial interest in semi-supervised learning (SSL) which aims to leverage unlabelled data to improve estimation or prediction. However, current SSL literature focuses primarily on settings where labelled data are selected uniformly at random from the population of interest. Stratified sampling, while posing additional analytical challenges, is highly applicable to many real-world problems. Moreover, no SSL methods currently exist for estimating the prediction performance of a fitted model when the labelled data are not selected uniformly at random. In this paper, we propose a two-step SSL procedure for evaluating a prediction rule derived from a working binary regression model based on the Brier score and overall misclassification rate under stratified sampling. In step I, we impute the missing labels via weighted regression with nonlinear basis functions to account for stratified sampling and to improve efficiency. In step II, we augment the initial imputations to ensure the consistency of the resulting estimators regardless of the specification of the prediction model or the imputation model. The final estimator is then obtained with the augmented imputations. We provide asymptotic theory and numerical studies illustrating that our proposals outperform their supervised counterparts in terms of efficiency gain. Our methods are motivated by electronic health record (EHR) research and validated with a real data analysis of an EHR-based study of diabetic neuropathy.<\/jats:p>","DOI":"10.1111\/rssb.12502","type":"journal-article","created":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T10:42:21Z","timestamp":1650969741000},"page":"1353-1391","source":"Crossref","is-referenced-by-count":13,"title":["Efficient Evaluation of Prediction Rules in Semi-Supervised Settings under Stratified Sampling"],"prefix":"10.1093","volume":"84","author":[{"given":"Jessica","family":"Gronsbell","sequence":"first","affiliation":[{"name":"Department of Statistical Sciences , Toronto , Ontario , Canada"},{"name":"University of Toronto , Toronto , Ontario , Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Molei","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Biostatistics , Boston , Massachusetts , USA"},{"name":"Harvard University , Boston , Massachusetts , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lu","family":"Tian","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Science , Palo Alto , California , USA"},{"name":"Stanford University , Palo Alto , California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianxi","family":"Cai","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics , Boston , Massachusetts , USA"},{"name":"Harvard Medical School , Boston , Massachusetts , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,4,26]]},"reference":[{"issue":"7","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1411","DOI":"10.1097\/MIB.0b013e31828133fd","article-title":"Improving case definition of crohn's disease and ulcerative colitis in electronic medical records using natural language processing: a novel informatics approach","volume":"19","author":"Ananthakrishnan","year":"2013","journal-title":"In Ammatory Bowel Diseases"},{"issue":"1\u20133","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1023\/B:MACH.0000033120.25363.1e","article-title":"Semi-supervised learning on riemannian manifolds","volume":"56","author":"Belkin","year":"2004","journal-title":"Machine Learning"},{"key":"2023030906301162700_","first-page":"2399","article-title":"Manifold regularization: a geometric framework for learning from labeled and unlabeled examples","volume":"7","author":"Belkin","year":"2006","journal-title":"The Journal of Machine Learning Research"},{"issue":"1","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1093\/biostatistics\/kxr021","article-title":"Evaluating prognostic accuracy of biomarkers in nested case\u2013control studies","volume":"13","author":"Cai","year":"2012","journal-title":"Biostatistics"},{"issue":"6","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"2102","DOI":"10.1109\/18.556600","article-title":"The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter","volume":"42","author":"Castelli","year":"1996","journal-title":"Information Theory, IEEE Transactions on"},{"issue":"4","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1541","DOI":"10.1214\/17-AOS1594","article-title":"Efficient and adaptive linear regression in semi-supervised settings","volume":"46","author":"Chakrabortty","year":"2018","journal-title":"The Annals of Statistics"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"542","DOI":"10.1109\/TNN.2009.2015974","article-title":"Semi-supervised learning (Chapelle, O. et al., eds.; 2006) [book reviews]","volume":"20","author":"Chapelle","year":"2009","journal-title":"IEEE Transactions on Neural Networks"},{"key":"2023030906301162700_","volume-title":"Stable mixing of complete and incomplete information","author":"Corduneanu","year":"2002"},{"key":"2023030906301162700_","volume-title":"Unlabeled data can degrade classification performance of generative classifiers","author":"Cozman","year":"2002"},{"key":"2023030906301162700_","volume-title":"Semi-supervised learning of mixture models","author":"Cozman","year":"2003"},{"issue":"382","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1080\/01621459.1983.10477973","article-title":"Estimating the error rate of a prediction rule: improvement on cross-validation","volume":"78","author":"Efron","year":"1983","journal-title":"Journal of the American Statistical Association"},{"issue":"394","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1080\/01621459.1986.10478291","article-title":"How biased is the apparent error rate of a prediction rule?","volume":"81","author":"Efron","year":"1986","journal-title":"Journal of the American Statistical Association"},{"issue":"438","key":"2023030906301162700_","first-page":"548","article-title":"Improvements on cross-validation: the 632+ bootstrap method","volume":"92","author":"Efron","year":"1997","journal-title":"Journal of the American Statistical Association"},{"issue":"9","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1979","DOI":"10.1093\/bioinformatics\/bti294","article-title":"Estimating misclassification error with small samples via bootstrap cross-validation","volume":"21","author":"Fu","year":"2005","journal-title":"Bioinformatics"},{"issue":"4","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1002\/bimj.200810443","article-title":"The performance of risk prediction models","volume":"50","author":"Gerds","year":"2008","journal-title":"Biometrical Journal: Journal of Mathematical Methods in Biosciences"},{"issue":"477","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"Gneiting","year":"2007","journal-title":"Journal of the American Statistical Association"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1111\/rssb.12264","article-title":"Semi-supervised approaches to efficient evaluation of model prediction performance","volume":"80","author":"Gronsbell","year":"2018","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2023030906301162700_","volume-title":"Construction and assessment of classification rules","author":"Hand","year":"1997"},{"issue":"1","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1111\/1467-9574.00153","article-title":"Measuring diagnostic accuracy of statistical prediction rules","volume":"55","author":"Hand","year":"2001","journal-title":"Statistica Neerlandica"},{"key":"2023030906301162700_","volume-title":"Exploiting generative models in discriminative classifiers","author":"Jaakkola","year":"1999"},{"issue":"29","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"5320","DOI":"10.1002\/sim.2968","article-title":"A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification","volume":"26","author":"Jiang","year":"2007","journal-title":"Statistics in Medicine"},{"issue":"2","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s10994-013-5329-8","article-title":"Semi-supervised learning with density-ratio estimation","volume":"91","author":"Kawakita","year":"2013","journal-title":"Machine Learning"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/j.neunet.2014.01.016","article-title":"Safe semi-supervised learning based on weighted likelihood","volume":"53","author":"Kawakita","year":"2014","journal-title":"Neural Networks"},{"issue":"6","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1038\/nrg2999","article-title":"Using electronic health records to drive discovery in disease genomics","volume":"12","author":"Kohane","year":"2011","journal-title":"Nature Reviews Genetics"},{"key":"2023030906301162700_","volume-title":"The curse of dimension in nonparametric regression","author":"Kpotufe","year":"2010"},{"key":"2023030906301162700_","volume-title":"Projected estimators for robust semi-supervised classification","author":"Krijthe","year":"2016"},{"issue":"8","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1002\/acr.20184","article-title":"Electronic medical records for discovery research in rheumatoid arthritis","volume":"62","author":"Liao","year":"2010","journal-title":"Arthritis Care & Research"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1002\/art.37801","article-title":"Autoantibodies, autoimmune risk alleles and clinical associations in rheumatoid arthritis cases and non-ra controls in the electronic medical records","volume":"65","author":"Liao","year":"2013","journal-title":"Arthritis and Rheumatism"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"h1885","DOI":"10.1136\/bmj.h1885","article-title":"Development of phenotype algorithms using electronic medical records and incorporating natural language processing","volume":"350","author":"Liao","year":"2015","journal-title":"BMJ"},{"issue":"4","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1219","DOI":"10.1111\/j.1541-0420.2012.01787.x","article-title":"Evaluating the predictive value of biomarkers with stratified case-cohort design","volume":"68","author":"Liu","year":"2012","journal-title":"Biometrics"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1007\/s10959-012-0454-z","article-title":"On edgeworth expansions in generalized urn models","volume":"27","author":"Mirakhmedov","year":"2014","journal-title":"Journal of Theoretical Probability"},{"issue":"15","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"3301","DOI":"10.1093\/bioinformatics\/bti499","article-title":"Prediction error estimation: a comparison of resampling methods","volume":"21","author":"Molinaro","year":"2005","journal-title":"Bioinformatics"},{"issue":"9","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1101\/gr.094615.109","article-title":"Instrumenting the health care enterprise for discovery research in the genomic era","volume":"19","author":"Murphy","year":"2009","journal-title":"Genome Research"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1093\/biomet\/asn027","article-title":"Optimal sampling and estimation strategies under the linear model","volume":"95","author":"Nedyalkova","year":"2008","journal-title":"Biometrika"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"2111","DOI":"10.1016\/S1573-4412(05)80005-4","article-title":"Large sample estimation and hypothesis testing","volume":"4","author":"Newey","year":"1994","journal-title":"Handbook of Econometrics"},{"issue":"4","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"558","DOI":"10.2307\/2342192","article-title":"On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection","volume":"97","author":"Neyman","year":"1934","journal-title":"Journal of the Royal Statistical Society"},{"issue":"1","key":"2023030906301162700_","first-page":"1229","article-title":"Manifold regularization and semi-supervised learning: some theoretical analyses","volume":"14","author":"Niyogi","year":"2013","journal-title":"The Journal of Machine Learning Research"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","DOI":"10.1214\/cbms\/1462061091","volume-title":"Empirical processes: theory and applications","author":"Pollard","year":"1990"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"479","DOI":"10.2307\/2532304","article-title":"Estimating exposure effects by modelling the expectation of exposure conditional on confounders","author":"Robins","year":"1992","journal-title":"Biometrics"},{"issue":"427","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1080\/01621459.1994.10476818","article-title":"Estimation of regression coefficients when some regressors are not always observed","volume":"89","author":"Robins","year":"1994","journal-title":"Journal of the American statistical Association"},{"key":"2023030906301162700_","volume-title":"Model assisted survey sampling","author":"S\u00e4rndal","year":"2003"},{"issue":"11","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1007\/s00439-014-1466-9","article-title":"Improving the power of genetic association tests with imperfect phenotype derived from electronic medical records","volume":"133","author":"Sinnott","year":"2014","journal-title":"Human Genetics"},{"key":"2023030906301162700_","doi-asserted-by":"crossref","DOI":"10.1145\/1390156.1390280","volume-title":"The asymptotics of semi-supervised learning in discriminative probabilistic models","author":"Sokolovska","year":"2008"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1093\/biomet\/asq035","article-title":"Bounded, efficient and doubly robust estimation with inverse weighting","volume":"97","author":"Tan","year":"2010","journal-title":"Biometrika"},{"issue":"2","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1093\/biomet\/asm036","article-title":"Model evaluation based on the sampling distribution of estimated absolute prediction error","volume":"94","author":"Tian","year":"2007","journal-title":"Biometrika"},{"key":"2023030906301162700_","volume-title":"Asymptotic statistics","author":"Van der Vaart","year":"2000"},{"key":"2023030906301162700_","volume-title":"Statistical analysis of semi-supervised regression","author":"Wasserman","year":"2008"},{"issue":"3","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1038\/clpt.2010.260","article-title":"The emerging role of electronic medical records in pharmacogenomics","volume":"89","author":"Wilke","year":"2011","journal-title":"Clinical Pharmacology & Therapeutics"},{"issue":"11","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"e78927","DOI":"10.1371\/journal.pone.0078927","article-title":"Modeling disease severity in multiple sclerosis using electronic health records","volume":"8","author":"Xia","year":"2013","journal-title":"PloS One"},{"issue":"5","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1093\/jamia\/ocv034","article-title":"Toward high-throughput phenotyping: unbiased automated feature extraction and selection from knowledge sources","volume":"22","author":"Yu","year":"2015","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"5","key":"2023030906301162700_","doi-asserted-by":"crossref","first-page":"2538","DOI":"10.1214\/18-AOS1756","article-title":"Semi-supervised inference: general theory and estimation of means","volume":"47","author":"Zhang","year":"2019","journal-title":"Annals of Statistics"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12502","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12502","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/4\/1353\/49463030\/jrsssb_84_4_1353.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/4\/1353\/49463030\/jrsssb_84_4_1353.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,20]],"date-time":"2023-11-20T16:39:51Z","timestamp":1700498391000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/84\/4\/1353\/7073262"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,26]]},"references-count":51,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4,26]]},"published-print":{"date-parts":[[2022,9,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12502","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,9]]},"published":{"date-parts":[[2022,4,26]]}}}