{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T11:55:55Z","timestamp":1790682955715,"version":"4.1.0"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2019,7,2]],"date-time":"2019-07-02T00:00:00Z","timestamp":1562025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,9,1]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries is small with high confidence. In this paper, confidence upper bounds for the false discovery proportion are constructed, which are simultaneous over all rejection cut-offs. In particular, this allows the user to select a set of hypotheses post hoc such that the false discovery proportion lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen (2006) has developed an exact, permutation-based and computationally feasible method for obtaining simultaneous false discovery proportion bounds. We propose an exact method which uniformly improves that procedure. Further, we provide a generalization of the method that lets the user select the shape of the simultaneous confidence bounds; this gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as significance analysis of microarrays and the maxT method of Westfall &amp; Young (1993), are obtained as special cases.<\/jats:p>","DOI":"10.1093\/biomet\/asz021","type":"journal-article","created":{"date-parts":[[2019,3,25]],"date-time":"2019-03-25T12:16:23Z","timestamp":1553516183000},"page":"635-649","source":"Crossref","is-referenced-by-count":31,"title":["Permutation-based simultaneous confidence bounds for the false discovery proportion"],"prefix":"10.1093","volume":"106","author":[{"given":"J","family":"Hemerik","sequence":"first","affiliation":[{"name":"Department of Biostatistics, Oslo Centre for Biostatistics and Epidemiology, University of Oslo, Sognsvannsveien 9, Domus Medica, 0372 Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"A","family":"Solari","sequence":"additional","affiliation":[{"name":"Department of Economics, Management and Statistics, University of Milano-Bicocca, Piazza dell\u2019Ateneo Nuovo 1, 20126 Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J J","family":"Goeman","sequence":"additional","affiliation":[{"name":"Biomedical Data Sciences, Leiden University Medical Center, Einthovenweg 20, 2333 ZC Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,7,2]]},"reference":[{"key":"2019081209351946600_B1","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: A practical and powerful approach to multiple testing","volume":"57","author":"Benjamini,","year":"1995","journal-title":"J. R. Statist. Soc. B"},{"key":"2019081209351946600_B2","article-title":"Post hoc inference via joint family-wise error rate control","author":"Blanchard,","year":"2018"},{"key":"2019081209351946600_B3","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1214\/08-EJS180","article-title":"Two simple sufficient conditions for FDR control.","volume":"2","author":"Blanchard,","year":"2008","journal-title":"Electron. J. Statist."},{"key":"2019081209351946600_B4","author":"Budczies,","year":"2011","journal-title":"cancerdata: Development and Validation of Diagnostic Tests from High-Dimensional Molecular Data: Datasets"},{"key":"2019081209351946600_B5","doi-asserted-by":"crossref","first-page":"1362","DOI":"10.1016\/j.spl.2007.12.018","article-title":"Modified Simes critical values under independence.","volume":"78","author":"Cai,","year":"2008","journal-title":"Statist. Prob. Lett."},{"key":"2019081209351946600_B6","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1111\/j.1467-9469.2008.00633.x","article-title":"Generalized augmentation to control the false discovery exceedance in multiple testing.","volume":"36","author":"Farcomeni,","year":"2009","journal-title":"Scand. J. Statist."},{"key":"2019081209351946600_B7","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1214\/009053604000000283","article-title":"A stochastic process approach to false discovery control.","volume":"32","author":"Genovese,","year":"2004","journal-title":"Ann. Statist."},{"key":"2019081209351946600_B8","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1198\/016214506000000339","article-title":"Exceedance control of the false discovery proportion.","volume":"101","author":"Genovese,","year":"2006","journal-title":"J. Am. Statist. Assoc."},{"key":"2019081209351946600_B9","doi-asserted-by":"crossref","DOI":"10.1093\/biomet\/asz041","article-title":"Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing","author":"Goeman,","year":"2019","journal-title":"Biometrika"},{"key":"2019081209351946600_B10","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1214\/11-STS356","article-title":"Multiple testing for exploratory research.","volume":"26","author":"Goeman,","year":"2011","journal-title":"Statist. Sci."},{"key":"2019081209351946600_B11","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.1002\/sim.6082","article-title":"Multiple hypothesis testing in genomics.","volume":"33","author":"Goeman,","year":"2014","journal-title":"Statist. Med."},{"key":"2019081209351946600_B12","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1002\/bimj.201300258","article-title":"On generalized Simes critical constants.","volume":"56","author":"Gou,","year":"2014","journal-title":"Biomet. J."},{"key":"2019081209351946600_B13","doi-asserted-by":"crossref","first-page":"1070","DOI":"10.1214\/14-AOS1214","article-title":"Further results on controlling the false discovery proportion.","volume":"42","author":"Guo,","year":"2014","journal-title":"Ann. Statist."},{"key":"2019081209351946600_B14","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1007\/s11749-017-0571-1","article-title":"Exact testing with random permutations","volume":"27","author":"Hemerik,","year":"2018","journal-title":"Test"},{"key":"2019081209351946600_B15","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1111\/rssb.12238","article-title":"False discovery proportion estimation by permutations: Confidence for significance analysis of microarrays","volume":"80","author":"Hemerik,","year":"2018","journal-title":"J. R. Statist. Soc. B"},{"key":"2019081209351946600_B16","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure.","volume":"6","author":"Holm,","year":"1979","journal-title":"Scand. J. Statist."},{"key":"2019081209351946600_B17","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1002\/bimj.19830250502","article-title":"Tests of the overall hypothesis for arbitrary dependence structures.","volume":"25","author":"Hommel,","year":"1983","journal-title":"Biomet. Zeit."},{"key":"2019081209351946600_B18","doi-asserted-by":"crossref","first-page":"4428","DOI":"10.1002\/sim.2865","article-title":"An investigation of two multivariate permutation methods for controlling the false discovery proportion.","volume":"26","author":"Korn,","year":"2007","journal-title":"Statist. Med."},{"key":"2019081209351946600_B19","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1016\/S0378-3758(03)00211-8","article-title":"Controlling the number of false discoveries: Application to high-dimensional genomic data.","volume":"124","author":"Korn,","year":"2004","journal-title":"J. Statist. Plan. Infer."},{"key":"2019081209351946600_B20","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s11222-005-4789-5","article-title":"Rotation tests.","volume":"15","author":"Langsrud,","year":"2005","journal-title":"Statist. Comp."},{"key":"2019081209351946600_B21","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1007\/978-1-4614-1412-4_57","article-title":"Generalizations of the familywise error rate","author":"Lehmann,","year":"2012","journal-title":"Selected Works of E. L. Lehmann"},{"key":"2019081209351946600_B22","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1093\/biomet\/63.3.655","article-title":"On closed testing procedures with special reference to ordered analysis of variance.","volume":"63","author":"Marcus,","year":"1976","journal-title":"Biometrika"},{"key":"2019081209351946600_B23","doi-asserted-by":"crossref","first-page":"75","DOI":"10.2307\/2346816","article-title":"Barnard\u2019s Monte Carlo tests: How many simulations?","volume":"28","author":"Marriott,","year":"1979","journal-title":"Appl. Statist."},{"key":"2019081209351946600_B24","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1111\/j.1467-9469.2005.00488.x","article-title":"False discovery control for multiple tests of association under general dependence.","volume":"33","author":"Meinshausen,","year":"2006","journal-title":"Scand. J. Statist."},{"key":"2019081209351946600_B25","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1093\/biomet\/92.4.893","article-title":"Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures.","volume":"92","author":"Meinshausen,","year":"2005","journal-title":"Biometrika"},{"key":"2019081209351946600_B26","doi-asserted-by":"crossref","first-page":"3369","DOI":"10.1214\/11-AOS946","article-title":"Asymptotic optimality of the Westfall\u2013Young permutation procedure for multiple testing under dependence.","volume":"39","author":"Meinshausen,","year":"2011","journal-title":"Ann. Statist."},{"key":"2019081209351946600_B27","doi-asserted-by":"crossref","DOI":"10.1002\/9780470689516","author":"Pesarin,","year":"2010","journal-title":"Permutation Tests for Complex Data: Theory, Applications and Software"},{"key":"2019081209351946600_B28","article-title":"R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing","year":"2019"},{"key":"2019081209351946600_B29","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1093\/biomet\/93.3.742","article-title":"Simes\u2019 procedure is \u2018valid on average\u2019.","volume":"93","author":"R\u00f8dland,","year":"2006","journal-title":"Biometrika"},{"key":"2019081209351946600_B30","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.1080\/01621459.2012.712876","article-title":"Comment: FDP vs FDR and the effect of conditioning.","volume":"107","author":"Schwartzman,","year":"2012","journal-title":"J. Am. Statist. Assoc."},{"key":"2019081209351946600_B31","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1093\/biomet\/asq075","article-title":"The effect of correlation in false discovery rate estimation.","volume":"98","author":"Schwartzman,","year":"2011","journal-title":"Biometrika"},{"key":"2019081209351946600_B32","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1093\/biomet\/73.3.751","article-title":"An improved Bonferroni procedure for multiple tests of significance.","volume":"73","author":"Simes,","year":"1986","journal-title":"Biometrika"},{"key":"2019081209351946600_B33","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1111\/biom.12238","article-title":"Rotation-based multiple testing in the multivariate linear model.","volume":"70","author":"Solari,","year":"2014","journal-title":"Biometrics"},{"key":"2019081209351946600_B34","doi-asserted-by":"crossref","first-page":"5116","DOI":"10.1073\/pnas.091062498","article-title":"Significance analysis of microarrays applied to the ionizing radiation response.","volume":"98","author":"Tusher,","year":"2001","journal-title":"Proc. Nat. Acad. Sci."},{"key":"2019081209351946600_B35","article-title":"Multiple testing. Part III. Procedures for control of the generalized family-wise error rate and proportion of false positives","author":"van der Laan,","year":"2004"},{"key":"2019081209351946600_B36","author":"Westfall,","year":"1993","journal-title":"Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment"}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asz021\/28897527\/asz021.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/biomet\/article-pdf\/106\/3\/635\/29133489\/asz021.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T03:33:41Z","timestamp":1721100821000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/106\/3\/635\/5527339"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,2]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2019,7,2]]},"published-print":{"date-parts":[[2019,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asz021","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,9]]},"published":{"date-parts":[[2019,7,2]]}}}