{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T16:03:00Z","timestamp":1791475380293,"version":"4.3.4"},"reference-count":33,"publisher":"Institute of Mathematical Statistics","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2020,6,1]]},"abstract":"<jats:p>We follow a post hoc, \u201cuser-agnostic\u201d approach to false discovery control in a large-scale multiple testing framework, as introduced by Genovese and Wasserman [J. Amer. Statist. Assoc. 101 (2006) 1408\u20131417], Goeman and Solari [Statist. Sci. 26 (2011) 584\u2013597]: the statistical guarantee on the number of correct rejections must hold for any set of candidate items, possibly selected by the user after having seen the data. To this end, we introduce a novel point of view based on a family of reference rejection sets and a suitable criterion, namely the joint familywise error rate over that family (JER for short). First, we establish how to derive post hoc bounds from a given JER control and analyze some general properties of this approach. We then develop procedures for controlling the JER in the case where reference regions are $p$-value level sets. These procedures adapt to dependencies and to the unknown quantity of signal (via a step-down principle). We also show interesting connections to confidence envelopes of Meinshausen [Scand. J. Stat. 33 (2006) 227\u2013237]; Genovese and Wasserman [J. Amer. Statist. Assoc. 101 (2006) 1408\u20131417], the closed testing based approach of Goeman and Solari [Statist. Sci. 26 (2011) 584\u2013597] and to the higher criticism of Donoho and Jin [Ann. Statist. 32 (2004) 962\u2013994]. Our theoretical statements are supported by numerical experiments.<\/jats:p>","DOI":"10.1214\/19-aos1847","type":"journal-article","created":{"date-parts":[[2020,7,17]],"date-time":"2020-07-17T04:02:13Z","timestamp":1594958533000},"page":"1281-1303","source":"Crossref","is-referenced-by-count":41,"title":["Post hoc confidence bounds on false positives using reference families"],"prefix":"10.1214","volume":"48","author":[{"given":"Gilles","family":"Blanchard","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierre","family":"Neuvial","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Etienne","family":"Roquain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"5","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. <i>Ann. Statist.<\/i> <b>29<\/b> 1165\u20131188.","DOI":"10.1214\/aos\/1013699998"},{"key":"4","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. <i>J. Roy. Statist. Soc. Ser. B<\/i> <b>57<\/b> 289\u2013300.","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"23","doi-asserted-by":"publisher","unstructured":"Lee, J. D., Sun, D. L., Sun, Y. and Taylor, J. E. (2016). Exact post-selection inference, with application to the lasso. <i>Ann. Statist.<\/i> <b>44<\/b> 907\u2013927.","DOI":"10.1214\/15-AOS1371"},{"key":"17","doi-asserted-by":"publisher","unstructured":"Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. <i>Ann. Statist.<\/i> <b>32<\/b> 1035\u20131061.","DOI":"10.1214\/009053604000000283"},{"key":"12","doi-asserted-by":"publisher","unstructured":"Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. <i>Ann. Statist.<\/i> <b>32<\/b> 962\u2013994.","DOI":"10.1214\/009053604000000265"},{"key":"10","doi-asserted-by":"publisher","unstructured":"B\u00fchlmann, P. and Mandozzi, J. (2014). High-dimensional variable screening and bias in subsequent inference, with an empirical comparison. <i>Comput. Statist.<\/i> <b>29<\/b> 407\u2013430.","DOI":"10.1007\/s00180-013-0436-3"},{"key":"16","unstructured":"Fithian, W., Sun, D. and Taylor, J. (2014). Optimal inference after model selection. Preprint. Available at <a href=\"arXiv:1410.2597\">arXiv:1410.2597<\/a>."},{"key":"22","unstructured":"Katsevich, E. and Ramdas, A. (2018). Towards \u201csimultaneous selective inference\u201d: Post-hoc bounds on the false discovery proportion. Preprint. Available at <a href=\"arXiv:1803.06790\">arXiv:1803.06790<\/a>."},{"key":"30","doi-asserted-by":"publisher","unstructured":"Taylor, J. and Tibshirani, R. J. (2015). Statistical learning and selective inference. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>112<\/b> 7629\u20137634.","DOI":"10.1073\/pnas.1507583112"},{"key":"15","doi-asserted-by":"publisher","unstructured":"Fan, J., Han, X. and Gu, W. (2012). Estimating false discovery proportion under arbitrary covariance dependence. <i>J. Amer. Statist. Assoc.<\/i> <b>107<\/b> 1019\u20131035.","DOI":"10.1080\/01621459.2012.720478"},{"key":"33","doi-asserted-by":"crossref","unstructured":"Woo, C.-W., Krishnan, A. and Wager, T. D. (2014). Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations. <i>NeuroImage<\/i> <b>91<\/b> 412\u2013419.","DOI":"10.1016\/j.neuroimage.2013.12.058"},{"key":"7","doi-asserted-by":"publisher","unstructured":"Berk, R., Brown, L., Buja, A., Zhang, K. and Zhao, L. (2013). Valid post-selection inference. <i>Ann. Statist.<\/i> <b>41<\/b> 802\u2013837.","DOI":"10.1214\/12-AOS1077"},{"key":"6","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Yekutieli, D. (2005). False discovery rate-adjusted multiple confidence intervals for selected parameters. <i>J. Amer. Statist. Assoc.<\/i> <b>100<\/b> 71\u201393. With comments and a rejoinder by the authors.","DOI":"10.1198\/016214504000001907"},{"key":"3","doi-asserted-by":"publisher","unstructured":"Belloni, A., Chernozhukov, V. and Hansen, C. (2014). Inference on treatment effects after selection among high-dimensional controls. <i>Rev. Econ. Stud.<\/i> <b>81<\/b> 608\u2013650.","DOI":"10.1093\/restud\/rdt044"},{"key":"26","doi-asserted-by":"publisher","unstructured":"Meinshausen, N. and B\u00fchlmann, P. (2005). Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures. <i>Biometrika<\/i> <b>92<\/b> 893\u2013907.","DOI":"10.1093\/biomet\/92.4.893"},{"key":"29","doi-asserted-by":"publisher","unstructured":"Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. <i>Biometrika<\/i> <b>73<\/b> 751\u2013754.","DOI":"10.1093\/biomet\/73.3.751"},{"key":"28","doi-asserted-by":"publisher","unstructured":"Romano, J. P. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypothesis testing. <i>J. Amer. Statist. Assoc.<\/i> <b>100<\/b> 94\u2013108.","DOI":"10.1198\/016214504000000539"},{"key":"18","doi-asserted-by":"publisher","unstructured":"Genovese, C. R. and Wasserman, L. (2006). Exceedance control of the false discovery proportion. <i>J. Amer. Statist. Assoc.<\/i> <b>101<\/b> 1408\u20131417.","DOI":"10.1198\/016214506000000339"},{"key":"1","doi-asserted-by":"publisher","unstructured":"Arlot, S., Blanchard, G. and Roquain, E. (2010). Some nonasymptotic results on resampling in high dimension. II. Multiple tests. <i>Ann. Statist.<\/i> <b>38<\/b> 83\u201399.","DOI":"10.1214\/08-AOS668"},{"key":"8","unstructured":"Blanchard, G., Neuvial, P. and Roquain, E. (2019). R package sansSouci version 0.8.0. <a href=\"https:\/\/github.com\/pneuvial\/sanssouci\">https:\/\/github.com\/pneuvial\/sanssouci<\/a>."},{"key":"9","doi-asserted-by":"crossref","unstructured":"Blanchard, G., Neuvial, P. and Roquain, E. (2019). Supplement to \u201cPost hoc confidence bounds on false positives using reference families.\u201d <a href=\"https:\/\/doi.org\/10.1214\/19-AOS1847SUPP\">https:\/\/doi.org\/10.1214\/19-AOS1847SUPP<\/a>.","DOI":"10.1214\/19-AOS1847SUPP"},{"key":"11","doi-asserted-by":"publisher","unstructured":"Dezeure, R., B\u00fchlmann, P., Meier, L. and Meinshausen, N. (2015). High-dimensional inference: Confidence intervals, $p$-values and $\\mathtt{R}$-software ${\\mathtt{hdi}}$. <i>Statist. Sci.<\/i> <b>30<\/b> 533\u2013558.","DOI":"10.1214\/15-STS527"},{"key":"14","unstructured":"Durand, G., Blanchard, G., Neuvial, P. and Roquain, E. (2018). Post hoc false positive control for spatially structured hypotheses. Preprint. Available at <a href=\"arXiv:1807.01470\">arXiv:1807.01470<\/a>."},{"key":"19","doi-asserted-by":"publisher","unstructured":"Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. <i>Statist. Sci.<\/i> <b>26<\/b> 584\u2013597.","DOI":"10.1214\/11-STS356"},{"key":"20","doi-asserted-by":"publisher","unstructured":"Hemerik, J. and Goeman, J. J. (2018). False discovery proportion estimation by permutations: Confidence for significance analysis of microarrays. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>80<\/b> 137\u2013155.","DOI":"10.1111\/rssb.12238"},{"key":"24","doi-asserted-by":"crossref","unstructured":"Li, W. (2012). Volcano plots in analyzing differential expressions with mrna microarrays. <i>Journal of Bioinformatics and Computational Biology<\/i> <b>10<\/b> 1231003.","DOI":"10.1142\/S0219720012310038"},{"key":"25","doi-asserted-by":"publisher","unstructured":"Meinshausen, N. (2006). False discovery control for multiple tests of association under general dependence. <i>Scand. J. Stat.<\/i> <b>33<\/b> 227\u2013237.","DOI":"10.1111\/j.1467-9469.2005.00488.x"},{"key":"31","doi-asserted-by":"crossref","unstructured":"van der Laan, M. J., Dudoit, S. and Pollard, K. S. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. <i>Stat. Appl. Genet. Mol. Biol.<\/i> <b>3<\/b> Art. 15, 27.","DOI":"10.2202\/1544-6115.1042"},{"key":"27","unstructured":"R Core Team (2017). <i>R: A Language and Environment for Statistical Computing<\/i>. R Foundation for Statistical Computing, Vienna, Austria."},{"key":"13","doi-asserted-by":"crossref","unstructured":"Dudoit, S. and van der Laan, M. J. (2008). <i>Multiple Testing Procedures with Applications to Genomics<\/i>. <i>Springer Series in Statistics<\/i>. Springer, New York.","DOI":"10.1007\/978-0-387-49317-6"},{"key":"2","doi-asserted-by":"publisher","unstructured":"Bachoc, F., Preinerstorfer, D. and Steinberger, L. (2019). Uniformly valid confidence intervals post-model-selection. <i>Ann. Statist.<\/i> To appear.","DOI":"10.1214\/19-AOS1815"},{"key":"21","doi-asserted-by":"crossref","unstructured":"Hemerik, J., Solari, A. and Goeman, J. (2019). Permutation-based simultaneous confidence bounds for the false discovery proportion. <i>Biometrika<\/i>. To appear.","DOI":"10.1093\/biomet\/asz021"},{"key":"32","unstructured":"Westfall, P. H. and Young, S. S. (1993). <i>Resampling-Based Multiple Testing. Examples and Methods for $P$-Value Adjustment<\/i>. Wiley."}],"container-title":["The Annals of Statistics"],"original-title":[],"link":[{"URL":"https:\/\/projecteuclid.org\/download\/pdfview_1\/euclid.aos\/1594972818","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T15:46:00Z","timestamp":1791474360000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-48\/issue-3\/Post-hoc-confidence-bounds-on-false-positives-using-reference-families\/10.1214\/19-AOS1847.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,1]]},"references-count":33,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,6,1]]}},"URL":"https:\/\/doi.org\/10.1214\/19-aos1847","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,1]]}}}