{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:18:54Z","timestamp":1784179134149,"version":"3.55.0"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2012,2,1]],"date-time":"2012-02-01T00:00:00Z","timestamp":1328054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2012,2]]},"abstract":"<jats:p>\n            High-quality reusable test collections and formal statistical hypothesis testing together support a rigorous experimental environment for information retrieval research. But as Armstrong et al. [2009b] recently argued, global analysis of experiments suggests that there has actually been little real improvement in ad hoc retrieval effectiveness over time. We investigate this phenomenon in the context of simultaneous testing of many hypotheses using a fixed set of data. We argue that the most common approaches to significance testing ignore a great deal of information about the world. Taking into account even a fairly small amount of this information can lead to very different conclusions about systems than those that have appeared in published literature. We demonstrate how to model a set of IR experiments for analysis both mathematically and practically, and show that doing so can cause\n            <jats:italic>p<\/jats:italic>\n            -values from statistical hypothesis tests to increase by orders of magnitude. This has major consequences on the interpretation of experimental results using reusable test collections: it is very difficult to conclude that\n            <jats:italic>anything<\/jats:italic>\n            is significant once we have modeled many of the sources of randomness in experimental design and analysis.\n          <\/jats:p>","DOI":"10.1145\/2094072.2094076","type":"journal-article","created":{"date-parts":[[2012,3,6]],"date-time":"2012-03-06T13:18:22Z","timestamp":1331039902000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":105,"title":["Multiple testing in statistical analysis of systems-based information retrieval experiments"],"prefix":"10.1145","volume":"30","author":[{"given":"Benjamin A.","family":"Carterette","sequence":"first","affiliation":[{"name":"University of Delaware, Newark, DE"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2012,3,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572153"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1056397485"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Box G. E. P. 1979. Robustness in the strategy of scientific model building. In Robustness in Statistics R. Launer and G. Wilkinson Eds.  Box G. E. P. 1979. Robustness in the strategy of scientific model building. In Robustness in Statistics R. Launer and G. Wilkinson Eds.","DOI":"10.1016\/B978-0-12-438150-6.50018-2"},{"key":"e_1_2_1_5_1","unstructured":"Bretz F. Hothorn T. and Westfall P. 2010. Multiple Comparisons Using R 1st Ed. CRC Press.  Bretz F. Hothorn T. and Westfall P. 2010. Multiple Comparisons Using R 1st Ed. CRC Press."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345543"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/2040317.2040333"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321530"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148262"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277934"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Draper N. R. and Smith H. 1998. Applied Regression Analysis 3rd Ed. Wiley-Interscience.  Draper N. R. and Smith H. 1998. Applied Regression Analysis 3rd Ed. Wiley-Interscience.","DOI":"10.1002\/9781118625590"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Gelman A. Carlin J. B. Stern H. S. and Rubin D. B. 2004. Bayesian Data Analysis. Chapman & Hall\/CRC.  Gelman A. Carlin J. B. Stern H. S. and Rubin D. B. 2004. Bayesian Data Analysis. Chapman & Hall\/CRC.","DOI":"10.1201\/9780429258480"},{"key":"e_1_2_1_13_1","unstructured":"Gelman A. Hill J. and Yajima M. 2009. Why we (usually) don't have to worry about multiple comparisons. arXiv:0907.2578v1 {stat.AP}.  Gelman A. Hill J. and Yajima M. 2009. Why we (usually) don't have to worry about multiple comparisons. arXiv:0907.2578v1 {stat.AP}."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.200810425"},{"key":"e_1_2_1_15_1","volume-title":"Multiple Comparisons: Theory and Methods","author":"Hsu J.","year":"1996","unstructured":"Hsu , J. 1996 . Multiple Comparisons: Theory and Methods 1 st Ed. Chapman and Hall\/CRC. Hsu, J. 1996. Multiple Comparisons: Theory and Methods 1st Ed. Chapman and Hall\/CRC.","edition":"1"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/160688.160758"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.294.2.218"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_1_19_1","unstructured":"Mardia K. V. Kent J. T. and Bibby J. M. 1980. Multivariate Analysis 1st Ed. Academic Press.  Mardia K. V. Kent J. T. and Bibby J. M. 1980. Multivariate Analysis 1st Ed. Academic Press."},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","unstructured":"McCullagh P. and Nelder J. 1989. Generalized Linear Models 2nd Ed. Chapman and Hall\/CRC.  McCullagh P. and Nelder J. 1989. Generalized Linear Models 2nd Ed. Chapman and Hall\/CRC.","DOI":"10.1007\/978-1-4899-3242-6"},{"key":"e_1_2_1_21_1","volume-title":"Simultaneous Statistical Inference","author":"Miller R. G.","unstructured":"Miller , R. G. 1981. Simultaneous Statistical Inference 2 nd Ed. Springer . Miller, R. G. 1981. Simultaneous Statistical Inference 2nd Ed. Springer.","edition":"2"},{"key":"e_1_2_1_22_1","volume-title":"A Primer on Linear Models","author":"Monahan J. F.","unstructured":"Monahan , J. F. 2008. A Primer on Linear Models 1 st Ed. Chapman and Hall\/CRC. Monahan, J. F. 2008. A Primer on Linear Models 1st Ed. Chapman and Hall\/CRC.","edition":"1"},{"key":"e_1_2_1_23_1","volume-title":"Multivariate Permutation Tests: With Applications in Biostatistics","author":"Pesarin F.","unstructured":"Pesarin , F. 2001. Multivariate Permutation Tests: With Applications in Biostatistics 1 st Ed. Wiley . Pesarin, F. 2001. Multivariate Permutation Tests: With Applications in Biostatistics 1st Ed. Wiley.","edition":"1"},{"key":"e_1_2_1_24_1","unstructured":"Raudenbush S. W. and Bryk A. S. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods 2nd Ed. Sage.  Raudenbush S. W. and Bryk A. S. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods 2nd Ed. Sage."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00027-7"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572050"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 3rd Text REtrieval Conference (TREC). 385--399","author":"Tague-Sutcliffe J.","unstructured":"Tague-Sutcliffe , J. and Blustein , J . 1994. A statistical analysis of the TREC-3 data . In Proceedings of the 3rd Text REtrieval Conference (TREC). 385--399 . Tague-Sutcliffe, J. and Blustein, J. 1994. A statistical analysis of the TREC-3 data. In Proceedings of the 3rd Text REtrieval Conference (TREC). 385--399."},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Venables W. N. and Ripley B. D. 2002. Modern Applied Statistics with S 4th Ed. Springer.   Venables W. N. and Ripley B. D. 2002. Modern Applied Statistics with S 4th Ed. Springer.","DOI":"10.1007\/978-0-387-21706-2"},{"key":"e_1_2_1_31_1","volume-title":"Overview of the 8th Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24","author":"Voorhees E. M.","unstructured":"Voorhees , E. M. and Harman , D . 1999 . Overview of the 8th Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24 . NIST Special Publication 500-246. Voorhees, E. M. and Harman, D. 1999. Overview of the 8th Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24. NIST Special Publication 500-246."},{"key":"e_1_2_1_32_1","volume-title":"Eds","author":"Voorhees E. M.","year":"2005","unstructured":"Voorhees , E. M. and Harman , D. K. , Eds . 2005 . TREC : Experiment and Evaluation in Information Retrieval. MIT Press . Voorhees, E. M. and Harman, D. K., Eds. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT Press."},{"key":"e_1_2_1_33_1","volume-title":"Sequential Analysis","author":"Wald A.","unstructured":"Wald , A. 1947. Sequential Analysis . John Wiley , New York . Wald, A. 1947. Sequential Analysis. John Wiley, New York."},{"key":"e_1_2_1_34_1","volume-title":"All of Statistics","author":"Wasserman L.","unstructured":"Wasserman , L. 2003. All of Statistics . Springer . Wasserman, L. 2003. All of Statistics. Springer."},{"key":"e_1_2_1_35_1","volume-title":"All of Nonparametric Statistics","author":"Wasserman L.","unstructured":"Wasserman , L. 2006. All of Nonparametric Statistics . Springer . Wasserman, L. 2006. All of Nonparametric Statistics. Springer."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458158"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/016555159402000405"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2094072.2094076","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2094072.2094076","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:48:41Z","timestamp":1750240121000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2094072.2094076"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,2]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2012,2]]}},"alternative-id":["10.1145\/2094072.2094076"],"URL":"https:\/\/doi.org\/10.1145\/2094072.2094076","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,2]]},"assertion":[{"value":"2011-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-03-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}