{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T11:24:58Z","timestamp":1776252298785,"version":"3.50.1"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T00:00:00Z","timestamp":1695427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["DMS-1844481"],"award-info":[{"award-number":["DMS-1844481"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["DMS-2311024"],"award-info":[{"award-number":["DMS-2311024"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterising the fit of the model to the underlying conditional law of labels given the features vector (Y\u2223X), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law Y\u2223X and treats that as a black-box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form H0:E[Df(Bern(\u03b7(X))\u2016Bern(\u03b7^(X)))]\u2264\u03c4 where Df represents an f-divergence function, and \u03b7(x), \u03b7^(x), respectively, denote the true and an estimate likelihood for a feature vector x admitting a positive label. We propose a novel test, called Goodness-of-fit with Randomisation and Scoring Procedure (GRASP) for testing H0, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X GRASP designed for model-X settings where the joint distribution of the features vector is known. Model-X GRASP uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.<\/jats:p>","DOI":"10.1093\/jrsssb\/qkad106","type":"journal-article","created":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T12:07:27Z","timestamp":1695470847000},"page":"215-245","source":"Crossref","is-referenced-by-count":2,"title":["GRASP: a goodness-of-fit test for classification learning"],"prefix":"10.1093","volume":"86","author":[{"given":"Adel","family":"Javanmard","sequence":"first","affiliation":[{"name":"Data Sciences and Operations Department, University of Southern California , Los Angeles , USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammad","family":"Mehrabi","sequence":"additional","affiliation":[{"name":"Data Sciences and Operations Department, University of Southern California , Los Angeles , USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2023,9,23]]},"reference":[{"key":"2024092300011998800_qkad106-B1","author":"Akaike","year":"1998"},{"key":"2024092300011998800_qkad106-B1a","author":"Balakrishnan S., & Wasserman L.","year":"2019"},{"issue":"5","key":"2024092300011998800_qkad106-B2","doi-asserted-by":"crossref","first-page":"2055","DOI":"10.1214\/15-AOS1337","article-title":"Controlling the false discovery rate via knockoffs","volume":"43","author":"Barber","year":"2015","journal-title":"The Annals of Statistics"},{"issue":"3","key":"2024092300011998800_qkad106-B3","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1214\/19-AOS1852","article-title":"Robust inference with knockoffs","volume":"48","author":"Barber","year":"2020","journal-title":"The Annals of Statistics"},{"issue":"535","key":"2024092300011998800_qkad106-B4","doi-asserted-by":"crossref","first-page":"1413","DOI":"10.1080\/01621459.2020.1729163","article-title":"Metropolized knockoff sampling","volume":"116","author":"Bates","year":"2021","journal-title":"Journal of the American Statistical Association"},{"key":"2024092300011998800_qkad106-B5","author":"Bates","year":"2021"},{"issue":"2","key":"2024092300011998800_qkad106-B6","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1287\/mnsc.1120.1641","article-title":"Robust solutions of optimization problems affected by uncertain probabilities","volume":"59","author":"Ben-Tal","year":"2013","journal-title":"Management Science"},{"issue":"1","key":"2024092300011998800_qkad106-B7","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1111\/rssb.12340","article-title":"The conditional permutation test for independence while controlling for confounders","volume":"82","author":"Berrett","year":"2020","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2024092300011998800_qkad106-B8","author":"Birg\u00e9","year":"2001"},{"issue":"3","key":"2024092300011998800_qkad106-B9","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1111\/rssb.12265","article-title":"Panning for gold: \u2018Model-X\u2019 knockoffs for high dimensional controlled variable selection","volume":"80","author":"Cand\u00e8s","year":"2018","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2024092300011998800_qkad106-B10","author":"Dwork","year":"2015"},{"issue":"4","key":"2024092300011998800_qkad106-B11","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1145\/3051088","article-title":"Guilt-free data reuse","volume":"60","author":"Dwork","year":"2017","journal-title":"Communications of the ACM"},{"issue":"394","key":"2024092300011998800_qkad106-B12","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1080\/01621459.1986.10478291","article-title":"How biased is the apparent error rate of a prediction rule?","volume":"81","author":"Efron","year":"1986","journal-title":"Journal of the American Statistical Association"},{"issue":"467","key":"2024092300011998800_qkad106-B13","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1198\/016214504000000692","article-title":"The estimation of prediction error: Covariance penalties and cross-validation","volume":"99","author":"Efron","year":"2004","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"2024092300011998800_qkad106-B14","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1111\/j.2517-6161.1996.tb02086.x","article-title":"On assessing goodness of fit of generalized linear models to sparse data","volume":"58","author":"Farrington","year":"1996","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"issue":"1\u20132","key":"2024092300011998800_qkad106-B15","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1002\/nav.3800030109","article-title":"An algorithm for quadratic programming","volume":"3","author":"Frank","year":"1956","journal-title":"Naval Research Logistics Quarterly"},{"key":"2024092300011998800_qkad106-B16","article-title":"Generative adversarial nets","volume":"27","author":"Goodfellow","year":"2014","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024092300011998800_qkad106-B17","author":"Guo","year":"2017"},{"key":"2024092300011998800_qkad106-B18","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning: Data mining, inference, and prediction","author":"Hastie","year":"2009"},{"issue":"18","key":"2024092300011998800_qkad106-B19","doi-asserted-by":"crossref","first-page":"2723","DOI":"10.1002\/sim.1200","article-title":"Goodness-of-fit processes for logistic regression: Simulation results","volume":"21","author":"Hosmer","year":"2002","journal-title":"Statistics in Medicine"},{"issue":"10","key":"2024092300011998800_qkad106-B20","doi-asserted-by":"crossref","first-page":"1043","DOI":"10.1080\/03610928008827941","article-title":"Goodness of fit tests for the multiple logistic regression model","volume":"9","author":"Hosmer","year":"1980","journal-title":"Communications in Statistics-Theory and Methods"},{"key":"2024092300011998800_qkad106-B21","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-21580-8","volume-title":"Nonparametric goodness-of-fit testing under Gaussian models","author":"Ingster","year":"2003"},{"issue":"3","key":"2024092300011998800_qkad106-B22","doi-asserted-by":"crossref","first-page":"241","DOI":"10.3103\/S1066530709030041","article-title":"Minimax goodness-of-fit testing in multivariate nonparametric regression","volume":"18","author":"Ingster","year":"2009","journal-title":"Mathematical Methods of Statistics"},{"issue":"3","key":"2024092300011998800_qkad106-B23","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1111\/rssb.12371","article-title":"Goodness-of-fit testing in high dimensional generalized linear models","volume":"82","author":"Jankov\u00e1","year":"2020","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2024092300011998800_qkad106-B24","author":"Javanmard","year":"2021"},{"issue":"2","key":"2024092300011998800_qkad106-B25","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1214\/17-AOS1559","article-title":"Online rules for control of false discovery rate and false discovery exceedance","volume":"46","author":"Javanmard","year":"2018","journal-title":"The Annals of Statistics"},{"key":"2024092300011998800_qkad106-B26","article-title":"Verified uncertainty calibration","volume":"32","author":"Kumar","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024092300011998800_qkad106-B27","volume-title":"Testing statistical hypotheses","author":"Lann","year":"1959"},{"issue":"4","key":"2024092300011998800_qkad106-B28","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.2307\/2532385","article-title":"A goodness-of-fit test for binary regression models, based on smoothing methods","volume":"47","author":"Le Cessie","year":"1991","journal-title":"Biometrics"},{"key":"2024092300011998800_qkad106-B29","author":"Lee","year":"2022"},{"issue":"2","key":"2024092300011998800_qkad106-B30","doi-asserted-by":"crossref","first-page":"333","DOI":"10.2307\/3318439","article-title":"Minimax nonparametric hypothesis testing: The case of an inhomogeneous alternative","volume":"5","author":"Lepski","year":"1999","journal-title":"Bernoulli"},{"issue":"1","key":"2024092300011998800_qkad106-B31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.0006-341X.2002.00001.x","article-title":"Model-checking techniques based on cumulative residuals","volume":"58","author":"Lin","year":"2002","journal-title":"Biometrics"},{"issue":"1","key":"2024092300011998800_qkad106-B32","first-page":"87","article-title":"Some comments on Cp","volume":"42","author":"Mallows","year":"2000","journal-title":"Technometrics"},{"key":"2024092300011998800_qkad106-B33","author":"Mukherjee","year":"2018"},{"key":"2024092300011998800_qkad106-B34","author":"Naeini","year":"2015"},{"key":"2024092300011998800_qkad106-B35","article-title":"Stochastic gradient methods for distributionally robust optimization with f-divergences","volume":"29","author":"Namkoong","year":"2016","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"11","key":"2024092300011998800_qkad106-B36","doi-asserted-by":"crossref","first-page":"5847","DOI":"10.1109\/TIT.2010.2068870","article-title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","volume":"56","author":"Nguyen","year":"2010","journal-title":"IEEE Transactions on Information Theory"},{"key":"2024092300011998800_qkad106-B37","author":"Nixon","year":"2019"},{"key":"2024092300011998800_qkad106-B38","article-title":"f-gan: Training generative neural samplers using variational divergence minimization","volume":"29","author":"Nowozin","year":"2016","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"420","key":"2024092300011998800_qkad106-B39","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1080\/01621459.1992.10476271","article-title":"Normal goodness-of-fit tests for multinomial models with large degrees of freedom","volume":"87","author":"Osius","year":"1992","journal-title":"Journal of the American Statistical Association"},{"issue":"5","key":"2024092300011998800_qkad106-B40","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1002\/gepi.21799","article-title":"Novel statistical tools for management of public databases facilitate community-wide replicability and control of false discovery","volume":"38","author":"Rosset","year":"2014","journal-title":"Genetic Epidemiology"},{"key":"2024092300011998800_qkad106-B41","article-title":"From fixed-x to random-x regression: Bias-variance decompositions, covariance penalties, and prediction error estimation","author":"Rosset","year":"2019","journal-title":"Journal of the American Statistical Association"},{"key":"2024092300011998800_qkad106-B42","volume-title":"Global sensitivity analysis: The primer","author":"Saltelli","year":"2008"},{"issue":"1","key":"2024092300011998800_qkad106-B43","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1111\/rssb.12234","article-title":"Goodness-of-fit tests for high dimensional linear models","volume":"80","author":"Shah","year":"2018","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"4","key":"2024092300011998800_qkad106-B44","doi-asserted-by":"crossref","first-page":"2258","DOI":"10.1137\/16M1058297","article-title":"Distributionally robust stochastic programming","volume":"27","author":"Shapiro","year":"2017","journal-title":"SIAM Journal on Optimization"},{"issue":"414","key":"2024092300011998800_qkad106-B45","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1080\/01621459.1991.10475059","article-title":"A lack-of-fit test for the mean function in a generalized linear model","volume":"86","author":"Su","year":"1991","journal-title":"Journal of the American Statistical Association"},{"issue":"1","key":"2024092300011998800_qkad106-B46","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1080\/10618600.2021.1923520","article-title":"The holdout randomization test for feature selection in black box models","volume":"31","author":"Tansey","year":"2022","journal-title":"Journal of Computational and Graphical Statistics"},{"issue":"1","key":"2024092300011998800_qkad106-B47","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1093\/biomet\/67.1.250","article-title":"A note on a goodness-of-fit test for the logistic regression model","volume":"67","author":"Tsiatis","year":"1980","journal-title":"Biometrika"},{"key":"2024092300011998800_qkad106-B48","author":"Vaicenavicius","year":"2019"},{"issue":"529","key":"2024092300011998800_qkad106-B49","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1080\/01621459.2020.1727235","article-title":"Cross-validation, risk estimation, and model selection: Comment on a paper by Rosset and Tibshirani","volume":"115","author":"Wager","year":"2020","journal-title":"Journal of the American Statistical Association"},{"key":"2024092300011998800_qkad106-B50","article-title":"Calibration tests in multi-class classification: A unifying framework","volume":"32","author":"Widmann","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024092300011998800_qkad106-B51","article-title":"Is a classification procedure good enough?\u2014A goodness-of-fit assessment tool for classification learning","author":"Zhang","year":"2021","journal-title":"Journal of the American Statistical Association"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/1\/215\/56629932\/qkad106.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/1\/215\/56629932\/qkad106.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,23]],"date-time":"2024-09-23T00:01:36Z","timestamp":1727049696000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/86\/1\/215\/7281031"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,23]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,9,23]]},"published-print":{"date-parts":[[2024,2,8]]}},"URL":"https:\/\/doi.org\/10.1093\/jrsssb\/qkad106","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,2]]},"published":{"date-parts":[[2023,9,23]]}}}