{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T03:03:52Z","timestamp":1791515032289,"version":"4.3.4"},"reference-count":47,"publisher":"Institute of Mathematical Statistics","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2020,12,1]]},"abstract":"<jats:p>While traditional multiple testing procedures prohibit adaptive analysis choices made by users, Goeman and Solari (Statist. Sci. 26 (2011) 584\u2013597) proposed a simultaneous inference framework that allows users such flexibility while preserving high-probability bounds on the false discovery proportion (FDP) of the chosen set. In this paper, we propose a new class of such simultaneous FDP bounds, tailored for nested sequences of rejection sets. While most existing simultaneous FDP bounds are based on closed testing using global null tests based on sorted $p$-values, we additionally consider the setting where side information can be leveraged to boost power, the variable selection setting where knockoff statistics can be used to order variables, and the online setting where decisions about rejections must be made as data arrives. Our finite-sample, closed form bounds are based on repurposing the FDP estimates from false discovery rate (FDR) controlling procedures designed for each of the above settings. These results establish a novel connection between the parallel literatures of simultaneous FDP bounds and FDR control methods, and use proof techniques employing martingales and filtrations that are new to both these literatures. We demonstrate the utility of our results by augmenting a recent knockoffs analysis of the UK Biobank dataset.<\/jats:p>","DOI":"10.1214\/19-aos1938","type":"journal-article","created":{"date-parts":[[2020,12,11]],"date-time":"2020-12-11T04:02:41Z","timestamp":1607659361000},"page":"3465-3487","source":"Crossref","is-referenced-by-count":33,"title":["Simultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings"],"prefix":"10.1214","volume":"48","author":[{"given":"Eugene","family":"Katsevich","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaditya","family":"Ramdas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"8","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. <i>Ann. Statist.<\/i> <b>29<\/b> 1165\u20131188.","DOI":"10.1214\/aos\/1013699998"},{"key":"6","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. <i>J. Roy. Statist. Soc. Ser. B<\/i> <b>57<\/b> 289\u2013300.","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"19","doi-asserted-by":"publisher","unstructured":"Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. <i>Ann. Statist.<\/i> <b>32<\/b> 1035\u20131061.","DOI":"10.1214\/009053604000000283"},{"key":"37","doi-asserted-by":"publisher","unstructured":"Owen, A. B. (2005). Variance of the number of false discoveries. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>67<\/b> 411\u2013426.","DOI":"10.1111\/j.1467-9868.2005.00509.x"},{"key":"44","doi-asserted-by":"publisher","unstructured":"Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>66<\/b> 187\u2013205.","DOI":"10.1111\/j.1467-9868.2004.00439.x"},{"key":"17","doi-asserted-by":"publisher","unstructured":"G\u2019Sell, M. G., Wager, S., Chouldechova, A. and Tibshirani, R. (2016). Sequential selection procedures and false discovery rate control. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>78<\/b> 423\u2013444.","DOI":"10.1111\/rssb.12122"},{"key":"4","doi-asserted-by":"publisher","unstructured":"Barber, R. F. and Cand\u00e8s, E. J. (2015). Controlling the false discovery rate via knockoffs. <i>Ann. Statist.<\/i> <b>43<\/b> 2055\u20132085.","DOI":"10.1214\/15-AOS1337"},{"key":"33","doi-asserted-by":"crossref","unstructured":"Li, A. and Barber, R. F. (2017). Accumulation tests for FDR control in ordered hypothesis testing. <i>J. Amer. Statist. Assoc.<\/i> <b>112<\/b> 837\u2013849.","DOI":"10.1080\/01621459.2016.1180989"},{"key":"36","doi-asserted-by":"publisher","unstructured":"Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. <i>Ann. Statist.<\/i> <b>34<\/b> 373\u2013393.","DOI":"10.1214\/009053605000000741"},{"key":"9","doi-asserted-by":"publisher","unstructured":"Berk, R., Brown, L., Buja, A., Zhang, K. and Zhao, L. (2013). Valid post-selection inference. <i>Ann. Statist.<\/i> <b>41<\/b> 802\u2013837.","DOI":"10.1214\/12-AOS1077"},{"key":"15","doi-asserted-by":"crossref","unstructured":"Efron, B. (2010). <i>Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction<\/i>. <i>Institute of Mathematical Statistics (IMS) Monographs<\/i> <b>1<\/b>. Cambridge Univ. Press, Cambridge.","DOI":"10.1017\/CBO9780511761362"},{"key":"18","doi-asserted-by":"publisher","unstructured":"Gavrilov, Y., Benjamini, Y. and Sarkar, S. K. (2009). An adaptive step-down procedure with proven FDR control under independence. <i>Ann. Statist.<\/i> <b>37<\/b> 619\u2013629.","DOI":"10.1214\/07-AOS586"},{"key":"47","doi-asserted-by":"publisher","unstructured":"Wellner, J. A. (1978). Limit theorems for the ratio of the empirical distribution function to the true distribution function. <i>Z. Wahrsch. Verw. Gebiete<\/i> <b>45<\/b> 73\u201388.","DOI":"10.1007\/BF00635964"},{"key":"7","doi-asserted-by":"publisher","unstructured":"Benjamini, Y. and Liu, W. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. <i>J. Statist. Plann. Inference<\/i> <b>82<\/b> 163\u2013170.","DOI":"10.1016\/S0378-3758(99)00040-3"},{"key":"29","doi-asserted-by":"publisher","unstructured":"Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. <i>Ann. Statist.<\/i> <b>33<\/b> 1138\u20131154.","DOI":"10.1214\/009053605000000084"},{"key":"20","doi-asserted-by":"publisher","unstructured":"Genovese, C. R. and Wasserman, L. (2006). Exceedance control of the false discovery proportion. <i>J. Amer. Statist. Assoc.<\/i> <b>101<\/b> 1408\u20131417.","DOI":"10.1198\/016214506000000339"},{"key":"22","doi-asserted-by":"publisher","unstructured":"Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. <i>Statist. Sci.<\/i> <b>26<\/b> 584\u2013597.","DOI":"10.1214\/11-STS356"},{"key":"35","doi-asserted-by":"publisher","unstructured":"Meinshausen, N. (2006). False discovery control for multiple tests of association under general dependence. <i>Scand. J. Stat.<\/i> <b>33<\/b> 227\u2013237.","DOI":"10.1111\/j.1467-9469.2005.00488.x"},{"key":"12","doi-asserted-by":"publisher","unstructured":"Cand\u00e8s, E., Fan, Y., Janson, L. and Lv, J. (2018). Panning for gold: \u2018model-X\u2019 knockoffs for high dimensional controlled variable selection. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>80<\/b> 551\u2013577.","DOI":"10.1111\/rssb.12265"},{"key":"42","doi-asserted-by":"publisher","unstructured":"Sesia, M., Sabatti, C. and Cand\u00e8s, E. J. (2019). Gene hunting with hidden Markov model knockoffs. <i>Biometrika<\/i> <b>106<\/b> 1\u201318.","DOI":"10.1093\/biomet\/asy033"},{"key":"1","doi-asserted-by":"crossref","unstructured":"Aharoni, E. and Rosset, S. (2014). Generalized $\\alpha$-investing: Definitions, optimality results and application to public databases. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>76<\/b> 771\u2013794.","DOI":"10.1111\/rssb.12048"},{"key":"2","doi-asserted-by":"crossref","unstructured":"Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S. et al. (2000). Gene ontology: Tool for the unification of biology. <i>Nat. Genet.<\/i> <b>25<\/b> 25.","DOI":"10.1038\/75556"},{"key":"3","doi-asserted-by":"publisher","unstructured":"Bachoc, F., Preinerstorfer, D. and Steinberger, L. (2016). Uniformly valid confidence intervals post-model-selection. Available at <a href=\"arXiv:1611.01043\">arXiv:1611.01043<\/a>.","DOI":"10.1214\/19-AOS1815"},{"key":"5","doi-asserted-by":"publisher","unstructured":"Barber, R. F. and Ramdas, A. (2017). The $p$-filter: Multilayer false discovery rate control for grouped hypotheses. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>79<\/b> 1247\u20131268.","DOI":"10.1111\/rssb.12218"},{"key":"10","doi-asserted-by":"publisher","unstructured":"Blanchard, G., Neuvial, P. and Roquain, E. (2020). Post hoc confidence bounds on false positives using reference families. <i>Ann. Statist.<\/i> <b>48<\/b> 1281\u20131303.","DOI":"10.1214\/19-AOS1847"},{"key":"11","doi-asserted-by":"crossref","unstructured":"Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O. et al. (2018). The UK Biobank resource with deep phenotyping and genomic data. <i>Nature<\/i> <b>562<\/b> 203.","DOI":"10.1038\/s41586-018-0579-z"},{"key":"13","doi-asserted-by":"publisher","unstructured":"Dvoretzky, A., Kiefer, J. and Wolfowitz, J. (1956). Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. <i>Ann. Math. Stat.<\/i> <b>27<\/b> 642\u2013669.","DOI":"10.1214\/aoms\/1177728174"},{"key":"14","unstructured":"Dwork, C., Su, W. J. and Zhang, L. (2018). Differentially private false discovery rate control. Available at <a href=\"arXiv:1807.04209v1\">arXiv:1807.04209v1<\/a>."},{"key":"16","doi-asserted-by":"publisher","unstructured":"Foster, D. P. and Stine, R. A. (2008). $\\alpha$-investing: A procedure for sequential control of expected false discoveries. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>70<\/b> 429\u2013444.","DOI":"10.1111\/j.1467-9868.2007.00643.x"},{"key":"21","unstructured":"Goeman, J., Hemerik, J. and Solari, A. (2019). Only closed testing procedures are admissible for controlling false discovery proportions. Available at <a href=\"arXiv:1901.04885\">arXiv:1901.04885<\/a>."},{"key":"23","doi-asserted-by":"publisher","unstructured":"Goeman, J. J., Meijer, R. J., Krebs, T. J. P. and Solari, A. (2019). Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing. <i>Biometrika<\/i> <b>106<\/b> 841\u2013856.","DOI":"10.1093\/biomet\/asz041"},{"key":"24","doi-asserted-by":"crossref","unstructured":"Hemerik, J., Solari, A. and Goeman, J. J. (2019). Permutation-based simultaneous confidence bounds for the false discovery proportion. <i>Biometrika<\/i> <b>106<\/b> 635\u2013649.","DOI":"10.1093\/biomet\/asz021"},{"key":"25","doi-asserted-by":"publisher","unstructured":"Javanmard, A. and Montanari, A. (2018). Online rules for control of false discovery rate and false discovery exceedance. <i>Ann. Statist.<\/i> <b>46<\/b> 526\u2013554.","DOI":"10.1214\/17-AOS1559"},{"key":"26","doi-asserted-by":"crossref","unstructured":"Katsevich, E. and Ramdas, A. (2020). Supplement to \u201cSimultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings.\u201d <a href=\"https:\/\/doi.org\/10.1214\/19-AOS1938SUPP\">https:\/\/doi.org\/10.1214\/19-AOS1938SUPP<\/a>.","DOI":"10.1214\/19-AOS1938"},{"key":"27","doi-asserted-by":"publisher","unstructured":"Katsevich, E. and Sabatti, C. (2019). Multilayer knockoff filter: Controlled variable selection at multiple resolutions. <i>Ann. Appl. Stat.<\/i> <b>13<\/b> 1\u201333.","DOI":"10.1214\/18-AOAS1185"},{"key":"31","doi-asserted-by":"publisher","unstructured":"Lei, L. and Fithian, W. (2018). AdaPT: An interactive procedure for multiple testing with side information. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>80<\/b> 649\u2013679.","DOI":"10.1111\/rssb.12274"},{"key":"32","unstructured":"Lei, L., Ramdas, A. and Fithian, W. (2017). STAR: A general interactive framework for FDR control under structural constraints. Available at <a href=\"arXiv:1710.02776\">arXiv:1710.02776<\/a>."},{"key":"34","doi-asserted-by":"crossref","unstructured":"McLean, C. Y., Bristor, D., Hiller, M., Clarke, S. L., Schaar, B. T., Lowe, C. B., Wenger, A. M. and Bejerano, G. (2010). GREAT improves functional interpretation of cis-regulatory regions. <i>Nat. Biotechnol.<\/i> <b>28<\/b> 495\u2013501.","DOI":"10.1038\/nbt.1630"},{"key":"40","doi-asserted-by":"publisher","unstructured":"Ramdas, A. K., Barber, R. F., Wainwright, M. J. and Jordan, M. I. (2019). A unified treatment of multiple testing with prior knowledge using the $p$-filter. <i>Ann. Statist.<\/i> <b>47<\/b> 2790\u20132821.","DOI":"10.1214\/18-AOS1765"},{"key":"41","unstructured":"Robbins, H. (1954). A one-sided confidence interval for an unknown distribution function. <i>Ann. Math. Stat.<\/i> <b>25<\/b> 409."},{"key":"28","doi-asserted-by":"crossref","unstructured":"Kuchibhotla, A. K., Brown, L. D., Buja, A., Cai, J., George, E. I. and Zhao, L. (2020). Valid post-selection inference in model-free linear regression. <i>Ann. Statist.<\/i> To appear.","DOI":"10.1214\/19-AOS1917"},{"key":"30","unstructured":"Lei, L. and Fithian, W. (2016). Power of ordered hypothesis testing. In <i>International Conference on Machine Learning<\/i> 2924\u20132932."},{"key":"38","unstructured":"Ramdas, A., Yang, F., Wainwright, M. J. and Jordan, M. I. (2017). Online control of the false discovery rate with decaying memory. In <i>Advances in Neural Information Processing Systems<\/i>."},{"key":"39","unstructured":"Ramdas, A., Zrnic, T., Wainwright, M. and Jordan, M. (2018). SAFFRON: An adaptive algorithm for online control of the false discovery rate. In <i>Proceedings of the 35th International Conference on Machine Learning<\/i> 4286\u20134294."},{"key":"43","doi-asserted-by":"crossref","unstructured":"Sesia, M., Katsevich, E., Bates, S., Cand\u00e8s, E. and Sabatti, C. (2019). Multi-resolution localization of causal variants across the genome. <i>BioRxiv<\/i>.","DOI":"10.1101\/631390"},{"key":"45","doi-asserted-by":"crossref","unstructured":"van der Laan, M. J., Dudoit, S. and Pollard, K. S. (2004). Multiple testing. Part III. Procedures for control of the generalized family-wise error rate and proportion of false positives.","DOI":"10.2202\/1544-6115.1042"},{"key":"46","unstructured":"Ville, J. (1939). <i>Etude critique de la notion de collectif<\/i>. Gauthier-Villars, Paris."}],"container-title":["The Annals of Statistics"],"original-title":[],"link":[{"URL":"https:\/\/projecteuclid.org\/download\/pdfview_1\/euclid.aos\/1607677243","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T15:21:38Z","timestamp":1791472898000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-48\/issue-6\/Simultaneous-high-probability-bounds-on-the-false-discovery-proportion-in\/10.1214\/19-AOS1938.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,1]]},"references-count":47,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,12,1]]}},"URL":"https:\/\/doi.org\/10.1214\/19-aos1938","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,1]]}}}