{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T04:16:40Z","timestamp":1792210600380,"version":"4.4.0"},"reference-count":62,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["grant DMS-2210637"],"award-info":[{"award-number":["grant DMS-2210637"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>This paper presents a conformal inference method for out-of-distribution testing that leverages side information from labelled outliers, which are commonly underutilized or even discarded by conventional conformal p-values. This solution is practical and blends inductive and transductive inference strategies to adaptively weight conformal p-values, while also automatically leveraging the most powerful model from a collection of one-class and binary classifiers. Further, this approach leads to rigorous false discovery rate control in multiple testing when combined with a conditional calibration strategy. Extensive numerical simulations show that the proposed method outperforms existing approaches.<\/jats:p>","DOI":"10.1093\/jrsssb\/qkad138","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T21:28:39Z","timestamp":1704922119000},"page":"671-693","source":"Crossref","is-referenced-by-count":12,"title":["Integrative conformal <i>p<\/i>-values for out-of-distribution testing with labelled outliers"],"prefix":"10.1093","volume":"86","author":[{"given":"Ziyi","family":"Liang","sequence":"first","affiliation":[{"name":"Department of Mathematics, University of Southern California , Los Angeles, CA ,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9046-907X","authenticated-orcid":false,"given":"Matteo","family":"Sesia","sequence":"additional","affiliation":[{"name":"Department of Data Sciences and Operations, University of Southern California , Los Angeles, CA ,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenguang","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Management and Center for Data Science, Zhejiang University , Hangzhou ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,1,10]]},"reference":[{"key":"2025011001015394200_qkad138-B1","author":"Angelopoulos","year":"2022"},{"issue":"1","key":"2025011001015394200_qkad138-B2","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1214\/20-AOS1965","article-title":"Predictive inference with the jackknife+","volume":"49","author":"Barber","year":"2021","journal-title":"The Annals of Statistics"},{"key":"2025011001015394200_qkad138-B3","author":"Barber","year":"2022"},{"issue":"523","key":"2025011001015394200_qkad138-B4","doi-asserted-by":"crossref","first-page":"1172","DOI":"10.1080\/01621459.2017.1336443","article-title":"Weighted false discovery rate control in large-scale multiple testing","volume":"113","author":"Basu","year":"2018","journal-title":"Journal of the American Statistical Association"},{"issue":"6","key":"2025011001015394200_qkad138-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3478535","article-title":"Distribution-free, risk-controlling prediction sets","volume":"68","author":"Bates","year":"2021","journal-title":"Journal of the ACM (JACM)"},{"issue":"1","key":"2025011001015394200_qkad138-B6","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1214\/22-AOS2244","article-title":"Testing for outliers with conformal p-values","volume":"51","author":"Bates","year":"2023","journal-title":"The Annals of Statistics"},{"key":"2025011001015394200_qkad138-B7","author":"Beganovic","year":"2018"},{"issue":"1","key":"2025011001015394200_qkad138-B8","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: A practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"Journal of the Royal Statistical Society: Series B"},{"issue":"3","key":"2025011001015394200_qkad138-B9","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1111\/1467-9469.00072","article-title":"Multiple hypotheses testing with weights","volume":"24","author":"Benjamini","year":"1997","journal-title":"Scandinavian Journal of Statistics"},{"issue":"4","key":"2025011001015394200_qkad138-B10","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1214\/aos\/1013699998","article-title":"The control of the false discovery rate in multiple testing under dependency","volume":"29","author":"Benjamini","year":"2001","journal-title":"Annals of Statistics"},{"issue":"539","key":"2025011001015394200_qkad138-B11","first-page":"1","article-title":"LAWS: A locally adaptive weighting and screening approach to spatial multiple testing","volume":"117","author":"Cai","year":"2021","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"2025011001015394200_qkad138-B12","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1109\/TR.2014.2315911","article-title":"Probabilistic novelty detection with support vector machines","volume":"63","author":"Clifton","year":"2014","journal-title":"IEEE Transactions on Reliability"},{"key":"2025011001015394200_qkad138-B13","author":"Colombo","year":"2020"},{"key":"2025011001015394200_qkad138-B14","author":"Deng","year":"2009"},{"issue":"12","key":"2025011001015394200_qkad138-B15","doi-asserted-by":"crossref","first-page":"3490","DOI":"10.1016\/j.patcog.2013.05.022","article-title":"One class random forests","volume":"46","author":"D\u00e9sir","year":"2013","journal-title":"Pattern Recognition"},{"key":"2025011001015394200_qkad138-B16","first-page":"22380","article-title":"Training uncertainty-aware classifiers with conformalized deep learning","volume":"35","author":"Einbinder","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"43","key":"2025011001015394200_qkad138-B17","doi-asserted-by":"crossref","first-page":"e2204569119","DOI":"10.1073\/pnas.2204569119","article-title":"Conformal prediction under feedback covariate shift for biomolecular design","volume":"119","author":"Fannjiang","year":"2022","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"6","key":"2025011001015394200_qkad138-B18","doi-asserted-by":"crossref","first-page":"3091","DOI":"10.1214\/21-AOS2137","article-title":"Conditional calibration for false discovery rate control under dependence","volume":"50","author":"Fithian","year":"2022","journal-title":"The Annals of Statistics"},{"issue":"2","key":"2025011001015394200_qkad138-B19","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1093\/imaiai\/iaaa017","article-title":"The limits of distribution-free conditional predictive inference","volume":"10","author":"Foygel Barber","year":"2021","journal-title":"Information and Inference"},{"issue":"3","key":"2025011001015394200_qkad138-B20","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1093\/biomet\/93.3.509","article-title":"False discovery control with p-value weighting","volume":"93","author":"Genovese","year":"2006","journal-title":"Biometrika"},{"issue":"1","key":"2025011001015394200_qkad138-B21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00401706.1969.10490657","article-title":"Procedures for detecting outlying observations in samples","volume":"11","author":"Grubbs","year":"1969","journal-title":"Technometrics"},{"key":"2025011001015394200_qkad138-B22","doi-asserted-by":"crossref","first-page":"108496","DOI":"10.1016\/j.patcog.2021.108496","article-title":"Nested conformal prediction and quantile out-of-bag ensemble methods","volume":"127","author":"Gupta","year":"2022","journal-title":"Pattern Recognition"},{"key":"2025011001015394200_qkad138-B23","author":"Haroush","year":"2021"},{"key":"2025011001015394200_qkad138-B24","author":"Hautamaki","year":"2004"},{"key":"2025011001015394200_qkad138-B25","author":"He","year":"2016"},{"issue":"491","key":"2025011001015394200_qkad138-B26","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1198\/jasa.2010.tm09329","article-title":"False discovery rate control with groups","volume":"105","author":"Hu","year":"2010","journal-title":"Journal of the American Statistical Association"},{"issue":"4","key":"2025011001015394200_qkad138-B27","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1111\/rssb.12411","article-title":"Covariate powered cross-weighted multiple testing","volume":"83","author":"Ignatiadis","year":"2021","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"issue":"3","key":"2025011001015394200_qkad138-B28","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1017\/S026988891300043X","article-title":"One-class classification: Taxonomy of study and review of techniques","volume":"29","author":"Khan","year":"2014","journal-title":"The Knowledge Engineering Review"},{"key":"2025011001015394200_qkad138-B29","first-page":"4138","article-title":"Predictive inference is free with the jackknife+-after-bootstrap","volume":"33","author":"Kim","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"1\u20132","key":"2025011001015394200_qkad138-B30","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1007\/s10472-013-9381-7","article-title":"Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories","volume":"74","author":"Laxhammar","year":"2015","journal-title":"Annals of Mathematics and Artificial Intelligence"},{"issue":"523","key":"2025011001015394200_qkad138-B31","doi-asserted-by":"crossref","first-page":"1094","DOI":"10.1080\/01621459.2017.1307116","article-title":"Distribution-free predictive inference for regression","volume":"113","author":"Lei","year":"2018","journal-title":"Journal of the American Statistical Association"},{"issue":"4","key":"2025011001015394200_qkad138-B32","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1111\/rssb.12274","article-title":"AdaPT: An interactive procedure for multiple testing with side information","volume":"80","author":"Lei","year":"2018","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"issue":"518","key":"2025011001015394200_qkad138-B33","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1080\/01621459.2016.1180989","article-title":"Accumulation tests for FDR control in ordered hypothesis testing","volume":"112","author":"Li","year":"2017","journal-title":"Journal of the American Statistical Association"},{"key":"2025011001015394200_qkad138-B34","author":"Liang","year":"2022"},{"key":"2025011001015394200_qkad138-B35","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1016\/j.neucom.2019.07.113","article-title":"Efficient conformal predictor ensembles","volume":"397","author":"Linusson","year":"2020","journal-title":"Neurocomputing"},{"key":"2025011001015394200_qkad138-B36","author":"L\u00f6fstr\u00f6m","year":"2013"},{"issue":"12","key":"2025011001015394200_qkad138-B37","doi-asserted-by":"crossref","first-page":"2481","DOI":"10.1016\/j.sigpro.2003.07.018","article-title":"Novelty detection: A review-part 1: Statistical approaches","volume":"83","author":"Markou","year":"2003","journal-title":"Signal Processing"},{"key":"2025011001015394200_qkad138-B38","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.sigpro.2013.12.026","article-title":"A review of novelty detection","volume":"99","author":"Pimentel","year":"2014","journal-title":"Signal Processing"},{"key":"2025011001015394200_qkad138-B39","author":"Rava","year":"2021"},{"issue":"86","key":"2025011001015394200_qkad138-B40","first-page":"2831","article-title":"Neyman\u2013Pearson classification, convexity and stochastic constraints","volume":"12","author":"Rigollet","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"2025011001015394200_qkad138-B41"},{"issue":"2","key":"2025011001015394200_qkad138-B42","first-page":"4","article-title":"With malice toward none: Assessing uncertainty via equalized coverage","volume":"2","author":"Romano","year":"2020","journal-title":"Harvard Data Science Review"},{"key":"2025011001015394200_qkad138-B43","first-page":"3581","article-title":"Classification with valid and adaptive coverage","volume":"33","author":"Romano","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025011001015394200_qkad138-B44","first-page":"678","article-title":"Optimal weighting for false discovery rate control","volume":"3","author":"Roquain","year":"2008","journal-title":"Electronic Journal of Statistics"},{"key":"2025011001015394200_qkad138-B45","article-title":"Conformalized frequency estimation from sketched data","volume":"35","author":"Sesia","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025011001015394200_qkad138-B46","author":"Song","year":"2019"},{"issue":"1","key":"2025011001015394200_qkad138-B47","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1214\/22-AOS2250","article-title":"Conditional predictive inference for stable algorithms","volume":"51","author":"Steinberger","year":"2023","journal-title":"The Annals of Statistics"},{"issue":"3","key":"2025011001015394200_qkad138-B48","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1111\/1467-9868.00346","article-title":"A direct approach to false discovery rates","volume":"64","author":"Storey","year":"2002","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"issue":"1","key":"2025011001015394200_qkad138-B49","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1467-9868.2004.00439.x","article-title":"Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach","volume":"66","author":"Storey","year":"2004","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"key":"2025011001015394200_qkad138-B50","article-title":"Conformal prediction under covariate shift","volume":"32","author":"Tibshirani","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"56","key":"2025011001015394200_qkad138-B51","first-page":"3011","article-title":"A plug-in approach to Neyman\u2013Pearson classification","volume":"14","author":"Tong","year":"2013","journal-title":"Journal of Machine Learning Research"},{"issue":"2","key":"2025011001015394200_qkad138-B52","doi-asserted-by":"crossref","first-page":"eaao1659","DOI":"10.1126\/sciadv.aao1659","article-title":"Neyman\u2013Pearson classification algorithms and NP receiver operating characteristics","volume":"4","author":"Tong","year":"2018","journal-title":"Science Advances"},{"key":"2025011001015394200_qkad138-B53"},{"key":"2025011001015394200_qkad138-B54","author":"Vovk","year":"2013"},{"issue":"1\u20132","key":"2025011001015394200_qkad138-B55","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/s10472-013-9368-4","article-title":"Cross-conformal predictors","volume":"74","author":"Vovk","year":"2015","journal-title":"Annals of Mathematics and Artificial Intelligence"},{"key":"2025011001015394200_qkad138-B56","author":"Vovk","year":"1999"},{"key":"2025011001015394200_qkad138-B57","volume-title":"Algorithmic learning in a random world","author":"Vovk","year":"2005"},{"key":"2025011001015394200_qkad138-B58","author":"Vovk","year":"2003"},{"key":"2025011001015394200_qkad138-B59","author":"Vovk","year":"2018"},{"key":"2025011001015394200_qkad138-B60","author":"Xu","year":"2021"},{"issue":"3","key":"2025011001015394200_qkad138-B61","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/3446776","article-title":"Understanding deep learning (still) requires rethinking generalization","volume":"64","author":"Zhang","year":"2021","journal-title":"Communications of the ACM"},{"issue":"212","key":"2025011001015394200_qkad138-B62","first-page":"7469","article-title":"Neyman\u2013Pearson classification under high-dimensional settings","volume":"17","author":"Zhao","year":"2016","journal-title":"Journal of Machine Learning Research"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/3\/671\/58504432\/qkad138.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/3\/671\/58504432\/qkad138.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T01:02:13Z","timestamp":1736470933000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/86\/3\/671\/7515027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,10]]},"references-count":62,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,1,10]]},"published-print":{"date-parts":[[2024,7,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jrsssb\/qkad138","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,7]]},"published":{"date-parts":[[2024,1,10]]}}}