{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T05:12:25Z","timestamp":1790831545026,"version":"4.1.0"},"reference-count":66,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,4,3]],"date-time":"2023-04-03T00:00:00Z","timestamp":1680480000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"crossref","award":["639.072.412"],"award-info":[{"award-number":["639.072.412"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Sum-based global tests are highly popular in multiple hypothesis testing. In this paper, we propose a general closed testing procedure for sum tests, which provides lower confidence bounds for the proportion of true discoveries (TDPs), simultaneously over all subsets of hypotheses. These simultaneous inferences come for free, i.e., without any adjustment of the \u03b1-level, whenever a global test is used. Our method allows for an exploratory approach, as simultaneity ensures control of the TDP even when the subset of interest is selected post hoc. It adapts to the unknown joint distribution of the data through permutation testing. Any sum test may be employed, depending on the desired power properties. We present an iterative shortcut for the closed testing procedure, based on the branch and bound algorithm, which converges to the full closed testing results, often after few iterations; even if it is stopped early, it controls the TDP. We compare the properties of different choices for the sum test through simulations, then we illustrate the feasibility of the method for high-dimensional data on brain imaging and genomics data.<\/jats:p>","DOI":"10.1093\/jrsssb\/qkad019","type":"journal-article","created":{"date-parts":[[2023,4,3]],"date-time":"2023-04-03T09:06:42Z","timestamp":1680512802000},"page":"664-683","source":"Crossref","is-referenced-by-count":12,"title":["Permutation-based true discovery guarantee by sum tests"],"prefix":"10.1093","volume":"85","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6696-2390","authenticated-orcid":false,"given":"Anna","family":"Vesely","sequence":"first","affiliation":[{"name":"Institute for Statistics, University of Bremen , Bremen , Germany"},{"name":"Department of Statistical Sciences, University of Padua , Padua , Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3181-8078","authenticated-orcid":false,"given":"Livio","family":"Finos","sequence":"additional","affiliation":[{"name":"Department of Statistical Sciences, University of Padua , Padua , Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4283-0259","authenticated-orcid":false,"given":"Jelle J","family":"Goeman","sequence":"additional","affiliation":[{"name":"Department of Biomedical Data Sciences, Leiden University Medical Center , Leiden , The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,4,3]]},"reference":[{"key":"2023071211305398200_qkad019-B1","author":"Andreella","year":"2023"},{"issue":"5","key":"2023071211305398200_qkad019-B2","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1038\/ejhg.2011.236","article-title":"Use of the gamma method for self-contained gene-set analysis of SNP data","volume":"20","author":"Biernacka","year":"2012","journal-title":"European Journal of Human Genetics"},{"key":"2023071211305398200_qkad019-B3","doi-asserted-by":"crossref","first-page":"119492","DOI":"10.1016\/j.neuroimage.2022.119492","article-title":"Notip: Non-parametric true discovery proportion control for brain imaging","volume":"260","author":"Blain","year":"2022","journal-title":"NeuroImage"},{"issue":"3","key":"2023071211305398200_qkad019-B4","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.1214\/19-AOS1847","article-title":"Post hoc confidence bounds on false positives using reference families","volume":"48","author":"Blanchard","year":"2020","journal-title":"The Annals of Statistics"},{"key":"2023071211305398200_qkad019-B5","doi-asserted-by":"crossref","first-page":"32","DOI":"10.3389\/fgene.2014.00032","article-title":"A modified generalized fisher method for combining probabilities from dependent tests","volume":"5","author":"Dai","year":"2014","journal-title":"Frontiers in Genetics"},{"issue":"1","key":"2023071211305398200_qkad019-B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/14-STS506","article-title":"Higher criticism for large-scale inference, especially for rare and weak effects","volume":"30","author":"Donoho","year":"2015","journal-title":"Statistical Science"},{"issue":"4","key":"2023071211305398200_qkad019-B7","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1002\/gepi.10264","article-title":"Rank truncated product of p-values, with application to genomewide association scans","volume":"25","author":"Dudbridge","year":"2003","journal-title":"Genetic Epidemiology"},{"issue":"4","key":"2023071211305398200_qkad019-B8","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1093\/bib\/bbz074","article-title":"Simultaneous enrichment analysis of all possible gene-sets: Unifying self-contained and competitive methods","volume":"21","author":"Ebrahimpoor","year":"2020","journal-title":"Briefings in Bioinformatics"},{"issue":"2","key":"2023071211305398200_qkad019-B9","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1080\/00223980.1972.9924813","article-title":"An additive method for combining probability values from independent experiments","volume":"80","author":"Edgington","year":"1972","journal-title":"The Journal of Psychology"},{"issue":"4","key":"2023071211305398200_qkad019-B10","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1214\/088342304000000396","article-title":"Permutation methods: A basis for exact inference","volume":"19","author":"Ernst","year":"2004","journal-title":"Statistical Science"},{"key":"2023071211305398200_qkad019-B11","author":"Finos","year":"2003"},{"key":"2023071211305398200_qkad019-B12","volume-title":"Statistical methods for research workers","author":"Fisher","year":"1925"},{"key":"2023071211305398200_qkad019-B13","doi-asserted-by":"crossref","first-page":"57","DOI":"10.2307\/2844116","article-title":"\u2018The coefficient of racial likeness\u2019 and the future of craniometry","volume":"66","author":"Fisher","year":"1936","journal-title":"The Journal of the Royal Anthropological Institute of Great Britain and Ireland"},{"issue":"476","key":"2023071211305398200_qkad019-B14","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1198\/016214506000000339","article-title":"Exceedance control of the false discovery proportion","volume":"101","author":"Genovese","year":"2006","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"2023071211305398200_qkad019-B15","doi-asserted-by":"crossref","first-page":"1218","DOI":"10.1214\/20-AOS1999","article-title":"Only closed testing procedures are admissible for controlling false discovery proportions","volume":"49","author":"Goeman","year":"2021","journal-title":"The Annals of Statistics"},{"issue":"4","key":"2023071211305398200_qkad019-B16","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1093\/biomet\/asz041","article-title":"Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing","volume":"106","author":"Goeman","year":"2019","journal-title":"Biometrika"},{"issue":"6","key":"2023071211305398200_qkad019-B17","doi-asserted-by":"crossref","first-page":"3782","DOI":"10.1214\/10-AOS829","article-title":"The sequential rejection principle of familywise error control","volume":"38","author":"Goeman","year":"2010","journal-title":"The Annals of Statistics"},{"issue":"4","key":"2023071211305398200_qkad019-B18","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1214\/11-STS356","article-title":"Multiple testing for exploratory research","volume":"26","author":"Goeman","year":"2011","journal-title":"Statistical Science"},{"issue":"3","key":"2023071211305398200_qkad019-B19","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1111\/j.1467-9868.2006.00551.x","article-title":"Testing against a high dimensional alternative","volume":"68","author":"Goeman","year":"2006","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"4","key":"2023071211305398200_qkad019-B20","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1007\/s11749-017-0571-1","article-title":"Exact testing with random permutations","volume":"27","author":"Hemerik","year":"2018","journal-title":"TEST"},{"issue":"1","key":"2023071211305398200_qkad019-B21","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1111\/rssb.12238","article-title":"False discovery proportion estimation by permutations: Confidence for significance analysis of microarrays","volume":"80","author":"Hemerik","year":"2018","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"issue":"3","key":"2023071211305398200_qkad019-B22","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1111\/rssb.12369","article-title":"Robust testing in generalized linear models by sign flipping score contributions","volume":"82","author":"Hemerik","year":"2020","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"issue":"3","key":"2023071211305398200_qkad019-B23","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1093\/biomet\/asz021","article-title":"Permutation-based simultaneous confidence bounds for the false discovery proportion","volume":"106","author":"Hemerik","year":"2019","journal-title":"Biometrika"},{"issue":"2","key":"2023071211305398200_qkad019-B24","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1214\/aoms\/1177729436","article-title":"The large-sample power of tests based on permutations of observations","volume":"23","author":"Hoeffding","year":"1952","journal-title":"The Annals of Mathematical Statistics"},{"issue":"18","key":"2023071211305398200_qkad019-B25","doi-asserted-by":"crossref","first-page":"2244","DOI":"10.1093\/bioinformatics\/btl383","article-title":"To permute or not to permute","volume":"22","author":"Huang","year":"2006","journal-title":"Bioinformatics"},{"issue":"5","key":"2023071211305398200_qkad019-B26","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1038\/nn.2303","article-title":"Circular analysis in systems neuroscience: The dangers of double dipping","volume":"12","author":"Kriegeskorte","year":"2009","journal-title":"Nature Neuroscience"},{"issue":"1","key":"2023071211305398200_qkad019-B27","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1534\/genetics.111.130542","article-title":"Novel rank-based approaches for discovery and replication in genomewide association studies","volume":"189","author":"Kuo","year":"2011","journal-title":"Genetics"},{"issue":"1","key":"2023071211305398200_qkad019-B28","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1111\/j.1467-842X.1961.tb00058.x","article-title":"The combination of probabilities: An application of orthonormal functions","volume":"3","author":"Lancaster","year":"1961","journal-title":"Australian Journal of Statistics"},{"issue":"3","key":"2023071211305398200_qkad019-B29","doi-asserted-by":"crossref","first-page":"497","DOI":"10.2307\/1910129","article-title":"An automatic method of solving discrete programming problems","volume":"28","author":"Land","year":"1960","journal-title":"Econometrica"},{"issue":"1","key":"2023071211305398200_qkad019-B30","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s11222-005-4789-5","article-title":"Rotation tests","volume":"15","author":"Langsrud","year":"2005","journal-title":"Statistics and Computing"},{"issue":"2A","key":"2023071211305398200_qkad019-B31","doi-asserted-by":"crossref","first-page":"994","DOI":"10.1214\/10-AOAS393","article-title":"An adaptively weighted statistic for detecting differential gene expression when combining multiple transcriptomic studies","volume":"5","author":"Li","year":"2011","journal-title":"Annals of Applied Statistics"},{"issue":"4","key":"2023071211305398200_qkad019-B32","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1214\/09-STS282","article-title":"The statistical analysis of fMRI data","volume":"23","author":"Lindquist","year":"2008","journal-title":"Statistical Science"},{"key":"2023071211305398200_qkad019-B33","first-page":"1971","article-title":"On the combination of independent tests","volume":"3","author":"Liptak","year":"1958","journal-title":"Magyar Tudom\u00e1nyos Akad\u00e9mia. Matematikai Kuatat\u00f3 Intezetenek Kozlemenyei"},{"issue":"529","key":"2023071211305398200_qkad019-B34","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1080\/01621459.2018.1554485","article-title":"Cauchy combination test: A powerful test with analytic p-value calculation under arbitrary dependency structures","volume":"115","author":"Liu","year":"2020","journal-title":"Journal of the American Statistical Association"},{"issue":"3","key":"2023071211305398200_qkad019-B35","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.csda.2003.11.020","article-title":"A systematic comparison of methods for combining p-values from independent tests","volume":"47","author":"Loughin","year":"2004","journal-title":"Computational Statistics & Data Analysis"},{"issue":"3","key":"2023071211305398200_qkad019-B36","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1093\/biomet\/63.3.655","article-title":"On closed testing procedures with special reference to ordered analysis of variance","volume":"63","author":"Marcus","year":"1976","journal-title":"Biometrika"},{"issue":"5","key":"2023071211305398200_qkad019-B37","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1093\/bib\/bbv091","article-title":"Multiple testing of gene sets from gene ontology: Possibilities and pitfalls","volume":"17","author":"Meijer","year":"2016","journal-title":"Briefings in Bioinformatics"},{"issue":"2","key":"2023071211305398200_qkad019-B38","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1111\/j.1467-9469.2005.00488.x","article-title":"False discovery control for multiple tests of association under general dependence","volume":"33","author":"Meinshausen","year":"2006","journal-title":"Scandinavian Journal of Statistics"},{"issue":"1","key":"2023071211305398200_qkad019-B39","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1287\/opre.18.1.24","article-title":"Branch-and-bound methods: General formulation and properties","volume":"18","author":"Mitten","year":"1970","journal-title":"Operations Research"},{"issue":"2","key":"2023071211305398200_qkad019-B40","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1016\/j.neuroimage.2012.04.014","article-title":"Multiple testing corrections, nonparametric methods, and random field theory","volume":"62","author":"Nichols","year":"2012","journal-title":"NeuroImage"},{"issue":"3\u20134","key":"2023071211305398200_qkad019-B41","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1093\/biomet\/25.3-4.379","article-title":"On a method of determining whether a sample of size n supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random","volume":"25","author":"Pearson","year":"1933","journal-title":"Biometrika"},{"key":"2023071211305398200_qkad019-B42","author":"Pernet","year":"2019"},{"key":"2023071211305398200_qkad019-B43","volume-title":"Multivariate permutation tests: With applications in biostatistics","author":"Pesarin","year":"2001"},{"key":"2023071211305398200_qkad019-B44","doi-asserted-by":"crossref","DOI":"10.1002\/9780470689516","volume-title":"Permutation tests for complex data: Theory, applications and software","author":"Pesarin","year":"2010"},{"key":"2023071211305398200_qkad019-B45","author":"R Core Team","year":"2017"},{"key":"2023071211305398200_qkad019-B46","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1016\/j.neuroimage.2018.07.060","article-title":"All-resolutions inference for brain imaging","volume":"181","author":"Rosenblatt","year":"2018","journal-title":"NeuroImage"},{"issue":"5","key":"2023071211305398200_qkad019-B47","doi-asserted-by":"crossref","first-page":"e0125587","DOI":"10.1371\/journal.pone.0125587","article-title":"Simultaneous statistical inference for epigenetic data","volume":"10","author":"Schildknecht","year":"2015","journal-title":"PLoS One"},{"issue":"3","key":"2023071211305398200_qkad019-B48","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1093\/biomet\/73.3.751","article-title":"An improved Bonferroni procedure for multiple tests of significance","volume":"73","author":"Simes","year":"1986","journal-title":"Biometrika"},{"issue":"4","key":"2023071211305398200_qkad019-B49","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1111\/biom.12238","article-title":"Rotation-based multiple testing in the multivariate linear model","volume":"70","author":"Solari","year":"2014","journal-title":"Biometrics"},{"issue":"4","key":"2023071211305398200_qkad019-B50","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1089\/cmb.2008.0144","article-title":"Properties of balanced permutations","volume":"16","author":"Southworth","year":"2009","journal-title":"Journal of Computational Biology"},{"key":"2023071211305398200_qkad019-B51","first-page":"1","article-title":"Large-scale simultaneous inference under dependence","author":"Tian","year":"2022","journal-title":"Scandinavian Journal of Statistics"},{"key":"2023071211305398200_qkad019-B52","author":"University of Padua","year":"2017"},{"key":"2023071211305398200_qkad019-B53","author":"Vesely","year":"2021"},{"key":"2023071211305398200_qkad019-B54","author":"Vesely","year":"2021"},{"issue":"4","key":"2023071211305398200_qkad019-B55","doi-asserted-by":"crossref","first-page":"asaa027","DOI":"10.1093\/biomet\/asaa027","article-title":"Combining p-values via averaging","volume":"107","author":"Vovk","year":"2020","journal-title":"Biometrika"},{"issue":"3","key":"2023071211305398200_qkad019-B56","doi-asserted-by":"crossref","first-page":"1736","DOI":"10.1214\/20-AOS2020","article-title":"E-values: Calibration, combination and applications","volume":"49","author":"Vovk","year":"2021","journal-title":"The Annals of Statistics"},{"key":"2023071211305398200_qkad019-B57","volume-title":"Resampling-based multiple testing: Examples and methods for p-value adjustment","author":"Westfall","year":"1993"},{"issue":"4","key":"2023071211305398200_qkad019-B58","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1073\/pnas.1814092116","article-title":"The harmonic mean p-value for combining dependent tests","volume":"116","author":"Wilson","year":"2019","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"100","key":"2023071211305398200_qkad019-B59","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1016\/j.neuroimage.2014.01.060","article-title":"Permutation inference for the general linear model","volume":"92","author":"Winkler","year":"2014","journal-title":"NeuroImage"},{"issue":"11","key":"2023071211305398200_qkad019-B60","doi-asserted-by":"crossref","first-page":"1537","DOI":"10.1002\/sim.3569","article-title":"Choosing an optimal method to combine p-values","volume":"28","author":"Won","year":"2009","journal-title":"Statistics in Medicine"},{"key":"2023071211305398200_qkad019-B61","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1016\/j.neuroimage.2013.12.058","article-title":"Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations","volume":"33","author":"Woo","year":"2014","journal-title":"Neuroimage"},{"issue":"1","key":"2023071211305398200_qkad019-B62","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.ajhg.2011.05.029","article-title":"Rare-variant association testing for sequencing data with the sequence kernel association test","volume":"89","author":"Wu","year":"2011","journal-title":"The American Journal of Human Genetics"},{"issue":"8","key":"2023071211305398200_qkad019-B63","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1002\/gepi.20422","article-title":"Pathway analysis by adaptive combination of p-values","volume":"33","author":"Yu","year":"2009","journal-title":"Genetic Epidemiology"},{"issue":"3","key":"2023071211305398200_qkad019-B64","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1002\/pst.304","article-title":"Combining p-values in large-scale genomics experiments","volume":"6","author":"Zaykin","year":"2007","journal-title":"Pharmaceutical Statistics"},{"issue":"2","key":"2023071211305398200_qkad019-B65","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1002\/gepi.0042","article-title":"Truncated product method for combining p-values","volume":"22","author":"Zaykin","year":"2002","journal-title":"Genetic Epidemiology"},{"issue":"1","key":"2023071211305398200_qkad019-B66","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1214\/19-AOAS1302","article-title":"TFisher: A powerful truncation and weighting procedure for combining p-values","volume":"14","author":"Zhang","year":"2020","journal-title":"The Annals of Applied Statistics"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/85\/3\/664\/50859850\/qkad019.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/85\/3\/664\/50859850\/qkad019.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T11:34:32Z","timestamp":1689161672000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/85\/3\/664\/7099913"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,3]]},"references-count":66,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,4,3]]},"published-print":{"date-parts":[[2023,7,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jrsssb\/qkad019","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,7]]},"published":{"date-parts":[[2023,4,3]]}}}