{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T04:22:14Z","timestamp":1791174134191,"version":"4.1.0"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2020,6,11]],"date-time":"2020-06-11T00:00:00Z","timestamp":1591833600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,12,1]]},"abstract":"<jats:title>Summary<\/jats:title>\n               <jats:p>This paper proposes general methods for the problem of multiple testing of a single hypothesis, with a standard goal of combining a number of $p$-values without making any assumptions about their dependence structure. A result by R\u00fcschendorf (1982) and, independently, Meng (1993) implies that the $p$-values can be combined by scaling up their arithmetic mean by a factor of 2, and no smaller factor is sufficient in general. A similar result by Mattner about the geometric mean replaces 2 by e. Based on more recent developments in mathematical finance, specifically, robust risk aggregation techniques, we extend these results to generalized means; in particular, we show that $K$ \u00a0$p$-values can be combined by scaling up their harmonic mean by a factor of $\\log K$ asymptotically as $K$ tends to infinity. This leads to a generalized version of the Bonferroni\u2013Holm procedure. We also explore methods using weighted averages of $p$-values. Finally, we discuss the efficiency of various methods of combining $p$-values and how to choose a suitable method in light of data and prior information.<\/jats:p>","DOI":"10.1093\/biomet\/asaa027","type":"journal-article","created":{"date-parts":[[2020,3,20]],"date-time":"2020-03-20T20:10:37Z","timestamp":1584735037000},"page":"791-808","source":"Crossref","is-referenced-by-count":135,"title":["Combining <i>p<\/i>-values via averaging"],"prefix":"10.1093","volume":"107","author":[{"given":"Vladimir","family":"Vovk","sequence":"first","affiliation":[{"name":"Department of Computer Science, Royal Holloway, University of London, Egham, Surrey TW20 0EX, U.K"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruodu","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Statistics and Actuarial Science, University of Waterloo, 200 University Avenue West, Waterloo, N2L 3G1 Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,6,11]]},"reference":[{"key":"2020121116433873500_B1","first-page":"93","article-title":"Risk aggregation with dependence uncertainty","volume":"54","author":"Bernard,","year":"2014","journal-title":"Insurance: Math. Econ."},{"key":"2020121116433873500_B2","doi-asserted-by":"crossref","DOI":"10.1002\/9780470743386","author":"Borenstein,","year":"2009","journal-title":"Introduction to Meta-Analysis"},{"key":"2020121116433873500_B3","doi-asserted-by":"crossref","first-page":"987","DOI":"10.2307\/2529826","article-title":"A method for combining non-independent, one-sided tests of significance","volume":"31","author":"Brown,","year":"1975","journal-title":"Biometrics"},{"key":"2020121116433873500_B4","article-title":"Annotated bibliography of some papers on combining significances or p-values","author":"Cousins,","year":"2008","journal-title":"arXiv:"},{"key":"2020121116433873500_B5","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1007\/s00780-006-0005-5","article-title":"Bounds for functions of dependent risks","volume":"10","author":"Embrechts,","year":"2006","journal-title":"Finan. Stoch."},{"key":"2020121116433873500_B6","doi-asserted-by":"crossref","first-page":"763","DOI":"10.1007\/s00780-015-0273-z","article-title":"Aggregation-robustness and model uncertainty of regulatory risk measures","volume":"19","author":"Embrechts,","year":"2015","journal-title":"Finan. Stoch."},{"key":"2020121116433873500_B7","first-page":"30","article-title":"Combining independent tests of significance","volume":"2","author":"Fisher,","year":"1948","journal-title":"Am. Statistician"},{"key":"2020121116433873500_B8","doi-asserted-by":"crossref","DOI":"10.1515\/9783110218053","author":"F\u00f6llmer,","year":"2011","journal-title":"Stochastic Finance: An Introduction in Discrete Time"},{"key":"2020121116433873500_B9","doi-asserted-by":"crossref","first-page":"1726","DOI":"10.1214\/009117906000000278","article-title":"Influence and sharp-threshold theorems for monotonic measures","volume":"34","author":"Graham,","year":"2006","journal-title":"Ann. Prob."},{"key":"2020121116433873500_B10","author":"Gut,","year":"2013","journal-title":"Probability: A Graduate Course"},{"key":"2020121116433873500_B11","author":"Hardy,","year":"1952","journal-title":"Inequalities"},{"key":"2020121116433873500_B12","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm,","year":"1979","journal-title":"Scand. J. Statist."},{"key":"2020121116433873500_B13","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1002\/bimj.19830250502","article-title":"Tests of the overall hypothesis for arbitrary dependence structures","volume":"25","author":"Hommel,","year":"1983","journal-title":"Biomet. J."},{"key":"2020121116433873500_B14","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/BF01894765","article-title":"Multiple test procedures for arbitrary dependence structures","volume":"33","author":"Hommel,","year":"1986","journal-title":"Metrika"},{"key":"2020121116433873500_B15","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1093\/biomet\/75.2.383","article-title":"A stagewise rejective multiple test procedure based on a modified Bonferroni test","volume":"75","author":"Hommel,","year":"1988","journal-title":"Biometrika"},{"key":"2020121116433873500_B16","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1080\/03461238.2015.1012223","article-title":"General convex order on risk aggregation","volume":"2016","author":"Jakobsons,","year":"2016","journal-title":"Scand. Actuar. J."},{"key":"2020121116433873500_B17","first-page":"388","article-title":"Sur la notion de la moyenne","volume":"12","author":"Kolmogorov,","year":"1930","journal-title":"Atti della Reale Accademia Nazionale dei Lincei. Classe di scienze fisiche, matematiche, e naturali. Rendiconti Serie VI"},{"key":"2020121116433873500_B18","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/S0167-7152(02)00310-3","article-title":"Combining dependent $p$-values","volume":"60","author":"Kost,","year":"2002","journal-title":"Statist. Prob. Lett."},{"key":"2020121116433873500_B19","article-title":"STAR: A general interactive framework for FDR control under structural constraints","author":"Lei,","year":"2017","journal-title":"arXiv:"},{"key":"2020121116433873500_B20","first-page":"171","article-title":"On the combination of independent tests","volume":"3","author":"Lipt\u00e1k,","year":"1958","journal-title":"A Magyar Tudom\u00e1nyos Akad\u00e9mia Matematikai Kutat\u00f3 Int\u00e9zet\u00e9nek K\u00f6zlem\u00e9nyei"},{"key":"2020121116433873500_B21","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1137\/1126086","article-title":"Estimates for the distribution function of the sum of two random variables with given marginal distributions","volume":"26","author":"Makarov,","year":"1981","journal-title":"Theory Prob. Appl."},{"key":"2020121116433873500_B22","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1093\/biomet\/63.3.655","article-title":"On closed testing procedures with special reference to ordered analysis of variance","volume":"63","author":"Marcus,","year":"1976","journal-title":"Biometrika"},{"key":"2020121116433873500_B23","author":"Mattner,","year":"2012","journal-title":"Abstract Book of the Tenth German Probability and Statistics Days"},{"key":"2020121116433873500_B24","author":"McNeil,","year":"2015","journal-title":"Quantitative Risk Management: Concepts, Techniques and Tools"},{"key":"2020121116433873500_B25","doi-asserted-by":"crossref","first-page":"1142","DOI":"10.1214\/aos\/1176325622","article-title":"Posterior predictive $p$-values","volume":"22","author":"Meng,","year":"1993","journal-title":"Ann. Statist."},{"key":"2020121116433873500_B26","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1007\/BF01893608","article-title":"Berechnung des maximalen Signifikanzniveaus des Testes \u2018Lehne ${H}_0$ ab, wenn $k$ unter $n$ gegebenen Tests zur Ablehnung f\u00fchren\u2019","volume":"27","author":"Morgenstern,","year":"1980","journal-title":"Metrika"},{"key":"2020121116433873500_B27","author":"Oosterhoff,","year":"1976","journal-title":"Combination of One-Sided Statistical Tests"},{"key":"2020121116433873500_B28","doi-asserted-by":"crossref","first-page":"3867","DOI":"10.1214\/09-AOS697","article-title":"Karl Pearson\u2019s meta-analysis revisited","volume":"37","author":"Owen,","year":"2009","journal-title":"Ann. Statist."},{"key":"2020121116433873500_B29","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1093\/biomet\/25.3-4.379","article-title":"On a method of determining whether a sample of size $n$ supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random","volume":"25","author":"Pearson,","year":"1933","journal-title":"Biometrika"},{"key":"2020121116433873500_B30","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1239\/jap\/1363784423","article-title":"Sharp bounds for sums of dependent risks","volume":"50","author":"Puccetti,","year":"2013","journal-title":"J. Appl. Prob."},{"key":"2020121116433873500_B31","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1007\/BF02204362","article-title":"Das maximale Signifikanzniveau des Tests \u2018Lehne ${H}_0$ ab, wenn $k$ unter $n$ gegebenen Tests zur Ablehnung f\u00fchren\u2019","volume":"25","author":"R\u00fcger,","year":"1978","journal-title":"Metrika"},{"key":"2020121116433873500_B32","doi-asserted-by":"crossref","first-page":"623","DOI":"10.2307\/1426677","article-title":"Random variables with maximum sums","volume":"14","author":"R\u00fcschendorf,","year":"1982","journal-title":"Adv. Appl. Prob."},{"key":"2020121116433873500_B33","doi-asserted-by":"crossref","first-page":"3921","DOI":"10.1016\/j.jspi.2009.05.030","article-title":"On the distributional transform, Sklar\u2019s theorem, and the empirical copula process","volume":"139","author":"R\u00fcschendorf,","year":"2009","journal-title":"J. Statist. Plan. Infer."},{"key":"2020121116433873500_B34","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-33590-7","author":"R\u00fcschendorf,","year":"2013","journal-title":"Mathematical Risk Analysis: Dependence, Risk Bounds, Optimal Allocations and Portfolios"},{"key":"2020121116433873500_B35","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4612-4250-5","author":"Schervish,","year":"1995","journal-title":"Theory of Statistics"},{"key":"2020121116433873500_B36","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1198\/000313001300339950","article-title":"Calibration of $p$-values for testing precise null hypotheses","volume":"55","author":"Sellke,","year":"2001","journal-title":"Am. Statistician"},{"key":"2020121116433873500_B37","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1093\/biomet\/73.3.751","article-title":"An improved Bonferroni procedure for multiple tests of significance","volume":"73","author":"Simes,","year":"1986","journal-title":"Biometrika"},{"key":"2020121116433873500_B38","author":"Tippett,","year":"1931","journal-title":"The Methods of Statistics: An Introduction Mainly for Experimentalists"},{"key":"2020121116433873500_B39","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1287\/moor.2015.0755","article-title":"Joint mixability","volume":"41","author":"Wang,","year":"2016","journal-title":"Math. Oper. Res."},{"key":"2020121116433873500_B40","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1007\/s00780-012-0200-5","article-title":"Bounds for the sum of dependent risks and worst Value-at-Risk with monotone marginal densities","volume":"17","author":"Wang,","year":"2013","journal-title":"Finance Stoch."},{"key":"2020121116433873500_B41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00031305.2019.1583913","article-title":"Moving to a world beyond \u2018$p &lt; 0.05$\u2019","volume":"73","author":"Wasserstein,","year":"2019","journal-title":"Am. Statistician"},{"key":"2020121116433873500_B42","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1073\/pnas.1814092116","article-title":"The harmonic mean $p$-value for combining dependent tests","volume":"116","author":"Wilson,","year":"2019","journal-title":"Proc. Nat. Acad. Sci."}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/biomet\/article-pdf\/107\/4\/791\/34865831\/asaa027.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/biomet\/article-pdf\/107\/4\/791\/34865831\/asaa027.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,12]],"date-time":"2020-12-12T07:41:58Z","timestamp":1607758918000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/107\/4\/791\/5856302"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,11]]},"references-count":42,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,6,11]]},"published-print":{"date-parts":[[2020,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asaa027","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,12]]},"published":{"date-parts":[[2020,6,11]]}}}