{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T18:56:02Z","timestamp":1791658562980,"version":"4.3.4"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2019,9,23]],"date-time":"2019-09-23T00:00:00Z","timestamp":1569196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Dutch Research Council"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.<\/jats:p>","DOI":"10.1093\/biomet\/asz041","type":"journal-article","created":{"date-parts":[[2019,7,9]],"date-time":"2019-07-09T19:09:59Z","timestamp":1562699399000},"page":"841-856","source":"Crossref","is-referenced-by-count":42,"title":["Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing"],"prefix":"10.1093","volume":"106","author":[{"given":"Jelle J","family":"Goeman","sequence":"first","affiliation":[{"name":"Biomedical Data Sciences, Leiden University Medical Center, Einthovenweg 20, 2333 ZC Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosa J","family":"Meijer","sequence":"additional","affiliation":[{"name":"Data Research Office, Antes, Parnassia Group, Rotterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thijmen J P","family":"Krebs","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Mekelweg 4, CD Delft, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aldo","family":"Solari","sequence":"additional","affiliation":[{"name":"Department of Economics, Management and Statistics, University of Milano-Bicocca, Piazza dell\u2019Ateneo Nuovo 1, 20126 Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,9,23]]},"reference":[{"key":"2019122310293688700_B1","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1002\/sta4.128","article-title":"General Glivenko\u2013Cantelli theorems","volume":"5","author":"Athreya,","year":"2016","journal-title":"Stat."},{"key":"2019122310293688700_B2","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1080\/01621459.2014.958156","article-title":"The empirical distribution of a large number of correlated normal variables","volume":"110","author":"Azriel,","year":"2015","journal-title":"J. Am. Statist. Assoc."},{"key":"2019122310293688700_B3","first-page":"289","article-title":"Controlling the false discovery rate: A practical and powerful approach to multiple testing","volume":"57","author":"Benjamini,","year":"1995","journal-title":"J. R. Statist. Soc. B"},{"key":"2019122310293688700_B4","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1214\/aos\/1013699998","article-title":"The control of the false discovery rate in multiple testing under dependency","volume":"29","author":"Benjamini,","year":"2001","journal-title":"Ann. Statist."},{"key":"2019122310293688700_B5","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1214\/12-AOS1077","article-title":"Valid post-selection inference","volume":"41","author":"Berk,","year":"2013","journal-title":"Ann. Statist."},{"key":"2019122310293688700_B6","author":"Bi,","year":"2017","journal-title":"Inferactive data analysis"},{"key":"2019122310293688700_B7","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1093\/biomet\/asp006","article-title":"Optimal testing of multiple hypotheses with common effect direction","volume":"96","author":"Bittman,","year":"2009","journal-title":"Biometrika"},{"key":"2019122310293688700_B8","article-title":"Post hoc inference via joint family-wise error rate control","author":"Blanchard,","year":"2018"},{"key":"2019122310293688700_B9","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s10463-015-0539-4","article-title":"On the Simes inequality in elliptical models","volume":"69","author":"Bodnar,","year":"2017","journal-title":"Ann. Inst. Statist. Math."},{"key":"2019122310293688700_B10","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1198\/jasa.2010.tm08127","article-title":"Shortcuts for locally consonant closed test procedures","volume":"105","author":"Brannath,","year":"2010","journal-title":"J. Am. Statist. Assoc."},{"key":"2019122310293688700_B11","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1214\/009053607000000037","article-title":"On the performance of FDR control: Constraints and a partial solution","volume":"35","author":"Chi,","year":"2007","journal-title":"Ann. Statist."},{"key":"2019122310293688700_B12","doi-asserted-by":"crossref","first-page":"302","DOI":"10.3150\/14-BEJ659","article-title":"On empirical distribution function of high-dimensional Gaussian vector components with an application to multiple testing","volume":"22","author":"Delattre,","year":"2016","journal-title":"Bernoulli"},{"key":"2019122310293688700_B13","volume-title":"Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction","author":"Efron,","year":"2012"},{"key":"2019122310293688700_B14","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1007\/s00362-015-0725-8","article-title":"On the Simes test under dependence","volume":"58","author":"Finner,","year":"2017","journal-title":"Statist. Papers"},{"key":"2019122310293688700_B15","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1111\/1467-9868.00347","article-title":"Operating characteristics and extensions of the false discovery rate procedure","volume":"64","author":"Genovese,","year":"2002","journal-title":"J. R. Statist. Soc. B"},{"key":"2019122310293688700_B16","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1214\/009053604000000283","article-title":"A stochastic process approach to false discovery control","volume":"32","author":"Genovese,","year":"2004","journal-title":"Ann. Statist."},{"key":"2019122310293688700_B17","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1198\/016214506000000339","article-title":"Exceedance control of the false discovery proportion","volume":"101","author":"Genovese,","year":"2006","journal-title":"J. Am. Statist. Assoc."},{"key":"2019122310293688700_B18","author":"Goeman,","year":"2018","journal-title":"hommel: Methods for Closed Testing with Simes Inequality, in Particular Hommel\u2019s Method"},{"key":"2019122310293688700_B19","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1214\/11-STS356","article-title":"Multiple testing for exploratory research","volume":"26","author":"Goeman,","year":"2011","journal-title":"Statist. Sci."},{"key":"2019122310293688700_B20","doi-asserted-by":"crossref","DOI":"10.1214\/13-STS435","article-title":"Correction note","volume":"28","author":"Goeman,","year":"2013","journal-title":"Statist. Sci."},{"key":"2019122310293688700_B21","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.1002\/sim.6082","article-title":"Multiple hypothesis testing in genomics","volume":"33","author":"Goeman,","year":"2014","journal-title":"Statist. Med."},{"key":"2019122310293688700_B22","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1093\/biomet\/asu032","article-title":"A class of improved hybrid Hochberg\u2013Hommel type step-up multiple test procedures","volume":"101","author":"Gou,","year":"2014","journal-title":"Biometrika"},{"key":"2019122310293688700_B23","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1080\/19466315.2015.1004270","article-title":"Closed testing in pharmaceutical research: Historical and recent developments","volume":"7","author":"Henning,","year":"2015","journal-title":"Statist. Biopharm. Res."},{"key":"2019122310293688700_B24","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1093\/biomet\/75.4.800","article-title":"A sharper Bonferroni procedure for multiple tests of significance","volume":"75","author":"Hochberg,","year":"1988","journal-title":"Biometrika"},{"key":"2019122310293688700_B25","doi-asserted-by":"crossref","DOI":"10.1007\/BF01894765","article-title":"Multiple test procedures for arbitrary dependence structures","volume":"33","author":"Hommel,","year":"1986","journal-title":"Metrika"},{"key":"2019122310293688700_B26","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1093\/biomet\/75.2.383","article-title":"A stagewise rejective multiple test procedure based on a modified Bonferroni test","volume":"75","author":"Hommel,","year":"1988","journal-title":"Biometrika"},{"key":"2019122310293688700_B27","article-title":"Towards \u2018simultaneous selective inference\u2019: Post-hoc bounds on the false discovery proportion","author":"Katsevich,","year":"2018"},{"key":"2019122310293688700_B28","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1111\/j.1467-9868.2005.00515.x","article-title":"Estimating the proportion of true null hypotheses, with application to DNA microarray data","volume":"67","author":"Langaas,","year":"2005","journal-title":"J. R. Statist. Soc. B"},{"key":"2019122310293688700_B29","doi-asserted-by":"crossref","first-page":"1138","DOI":"10.1214\/009053605000000084","article-title":"Generalizations of the familywise error rate","volume":"33","author":"Lehmann,","year":"2005","journal-title":"Ann. Statist."},{"key":"2019122310293688700_B30","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1111\/rssb.12274","article-title":"AdaPT: An interactive procedure for multiple testing with side information","volume":"80","author":"Lei,","year":"2018","journal-title":"J. R. Statist. Soc. B"},{"key":"2019122310293688700_B31","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1093\/biomet\/63.3.655","article-title":"Closed testing procedures with special reference to ordered analysis of variance","volume":"63","author":"Marcus,","year":"1976","journal-title":"Biometrika"},{"key":"2019122310293688700_B32","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1002\/bimj.201700316","article-title":"Hommel\u2019s procedure in linear time","volume":"61","author":"Meijer,","year":"2019","journal-title":"Biomet. J."},{"key":"2019122310293688700_B33","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1111\/j.1467-9469.2005.00488.x","article-title":"False discovery control for multiple tests of association under general dependence","volume":"33","author":"Meinshausen,","year":"2006","journal-title":"Scand. J. Statist."},{"key":"2019122310293688700_B34","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1198\/108571106X129135","article-title":"Estimating the number of true null hypotheses from a histogram of $p$ values","volume":"11","author":"Nettleton,","year":"2006","journal-title":"J. Agric. Biol. Envir. Statist."},{"key":"2019122310293688700_B35","year":"2019","journal-title":"R: A Language and Environment for Statistical Computing"},{"key":"2019122310293688700_B36","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1093\/biomet\/93.3.742","article-title":"Simes\u2019 procedure is \u2018valid on average\u2019","volume":"93","author":"R\u00f8dland,","year":"2006","journal-title":"Biometrika"},{"key":"2019122310293688700_B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2202\/1557-4679.1300","article-title":"Consonance and the closure method in multiple testing","volume":"7","author":"Romano,","year":"2011","journal-title":"Int. J. Biostatist."},{"key":"2019122310293688700_B38","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1016\/j.neuroimage.2018.07.060","article-title":"All-resolutions inference for brain imaging","volume":"181","author":"Rosenblatt,","year":"2018","journal-title":"NeuroImage"},{"key":"2019122310293688700_B39","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1214\/193940307000000167","article-title":"On the Simes inequality and its generalization","volume-title":"Beyond Parametrics in Interdisciplinary Research: Festschrift in Honor of Professor Pranab K. Sen","author":"Sarkar,","year":"2008"},{"key":"2019122310293688700_B40","article-title":"The FDR-linking theorem","author":"Su,","year":"2018"},{"key":"2019122310293688700_B41","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1016\/j.jspi.2007.06.006","article-title":"False discovery rate control for non-positively regression dependent test statistics","volume":"138","author":"Yekutieli,","year":"2008","journal-title":"J. Statist. Plan. Infer."}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/biomet\/article-pdf\/106\/4\/841\/30646685\/asz041.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/biomet\/article-pdf\/106\/4\/841\/30646685\/asz041.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T11:13:55Z","timestamp":1663931635000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/106\/4\/841\/5572659"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,23]]},"references-count":41,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2019,9,23]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asz041","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,9,23]]}}}