{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T04:56:54Z","timestamp":1755838614075,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":35,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,8,7]],"date-time":"2017-08-07T00:00:00Z","timestamp":1502064000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,8,7]]},"DOI":"10.1145\/3077136.3080766","type":"proceedings-article","created":{"date-parts":[[2017,7,28]],"date-time":"2017-07-28T19:35:01Z","timestamp":1501270501000},"page":"25-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation"],"prefix":"10.1145","author":[{"given":"Tetsuya","family":"Sakai","sequence":"first","affiliation":[{"name":"Waseda University, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,8,7]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of TREC","author":"Aslam Javed","year":"2016","unstructured":"Javed Aslam , Fernando Diaz , Matthew Ekstrand-Abueg , Richard McCreadie , Virgil Pavlu , and Tetsuya Sakai 2016 . TREC 2015 Temporal Summarization Track . In Proceedings of TREC 2015. Javed Aslam, Fernando Diaz, Matthew Ekstrand-Abueg, Richard McCreadie, Virgil Pavlu, and Tetsuya Sakai 2016. TREC 2015 Temporal Summarization Track. In Proceedings of TREC 2015."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1098\/rstl.1763.0053"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23318-0_11"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_3_2_1_5_1","volume-title":"Bayesian Inference for Information Retrieval Evaluation Proceedings of ACM ICTIR","author":"Carterette Ben","year":"2015","unstructured":"Ben Carterette . 2015 . Bayesian Inference for Information Retrieval Evaluation Proceedings of ACM ICTIR 2015. 31--40. Ben Carterette. 2015. Bayesian Inference for Information Retrieval Evaluation Proceedings of ACM ICTIR 2015. 31--40."},{"key":"e_1_3_2_1_6_1","volume-title":"A Comparative Analysis of Cascade Measures for Novelty and Diversity Proceedings of ACM WSDM","author":"Clarke Charles L. A.","year":"2011","unstructured":"Charles L. A. Clarke , Nick Craswell , Ian Soboroff , and Azin Ashkan 2011. A Comparative Analysis of Cascade Measures for Novelty and Diversity Proceedings of ACM WSDM 2011 . 75--84. Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Azin Ashkan 2011. A Comparative Analysis of Cascade Measures for Novelty and Diversity Proceedings of ACM WSDM 2011. 75--84."},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of TREC","author":"Clarke Clarles L. A.","year":"2011","unstructured":"Clarles L. A. Clarke , Nick Craswell , Ian Soboroff , and Ellen M . Voorhees 2012. Overview of the TREC 2011 Web Track . In Proceedings of TREC 2011 . Clarles L. A. Clarke, Nick Craswell, Ian Soboroff, and Ellen M. Voorhees 2012. Overview of the TREC 2011 Web Track. In Proceedings of TREC 2011."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1198\/016214505000000033"},{"key":"e_1_3_2_1_9_1","volume-title":"Statistical Methods for Research Workers","author":"Fisher Ronald A.","unstructured":"Ronald A. Fisher . 1970. Statistical Methods for Research Workers ( 14 th Edition). Oliver & Boyd . Ronald A. Fisher. 1970. Statistical Methods for Research Workers (14th Edition). Oliver & Boyd.","edition":"14"},{"volume-title":"Inference and Monitoring Convergence. Markov Chan Monte Carlo in Practice","author":"Gelman Andrew","key":"e_1_3_2_1_10_1","unstructured":"Andrew Gelman . 1996. Inference and Monitoring Convergence. Markov Chan Monte Carlo in Practice , W. R. Gilks, S. Richardson, and D. J. Spiegelhalter (Eds.). Chapman & Hall\/CRC , Chapter 8. Andrew Gelman. 1996. Inference and Monitoring Convergence. Markov Chan Monte Carlo in Practice, W. R. Gilks, S. Richardson, and D. J. Spiegelhalter (Eds.). Chapman & Hall\/CRC, Chapter 8."},{"key":"e_1_3_2_1_11_1","unstructured":"Andrew Gelman Jennifer Hill and Masanao Yajima. 2008. Why We (Usually) Don't Have to Worry about Multiple Comparisons. Technical Report.  Andrew Gelman Jennifer Hill and Masanao Yajima. 2008. Why We (Usually) Don't Have to Worry about Multiple Comparisons. Technical Report."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Lisa L. Harlow Stanley A. Mulaik and James H. Steiger (Eds.). 2016. What If There Were No Significance Tests (Classic Edition). Routledge.  Lisa L. Harlow Stanley A. Mulaik and James H. Steiger (Eds.). 2016. What If There Were No Significance Tests (Classic Edition). Routledge.","DOI":"10.4324\/9781315629049"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0029146"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0029146"},{"volume-title":"Doing Bayesian Data Analysis","author":"Kruschke John K.","key":"e_1_3_2_1_15_1","unstructured":"John K. Kruschke . 2015. Doing Bayesian Data Analysis ( Second Edition). Elsevier . John K. Kruschke. 2015. Doing Bayesian Data Analysis (Second Edition). Elsevier."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1201\/b10905-6"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0149-x"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0149-x"},{"key":"e_1_3_2_1_19_1","unstructured":"Matia Okubo and Kensuke Okada 2012. Psychological Statistics to Tell Your Story: Effect Size Confidence Interval and Power. Keiso Shobo.  Matia Okubo and Kensuke Okada 2012. Psychological Statistics to Tell Your Story: Effect Size Confidence Interval and Power. Keiso Shobo."},{"key":"e_1_3_2_1_20_1","volume-title":"Evaluating Evaluation Metrics based on the Bootstrap Proceedings of ACM SIGIR","author":"Sakai Tetsuya","year":"2006","unstructured":"Tetsuya Sakai . 2006. Evaluating Evaluation Metrics based on the Bootstrap Proceedings of ACM SIGIR 2006 . 525--532. Tetsuya Sakai. 2006. Evaluating Evaluation Metrics based on the Bootstrap Proceedings of ACM SIGIR 2006. 525--532."},{"key":"e_1_3_2_1_21_1","volume-title":"Evaluating Diversified Search Results Using Per-Intent Graded Relevance Proceedings of ACM SIGIR","author":"Sakai Tetsuya","year":"2011","unstructured":"Tetsuya Sakai . 2011 . Evaluating Diversified Search Results Using Per-Intent Graded Relevance Proceedings of ACM SIGIR 2011. 1043--1052. Tetsuya Sakai. 2011. Evaluating Diversified Search Results Using Per-Intent Graded Relevance Proceedings of ACM SIGIR 2011. 1043--1052."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2641383.2641385"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914684"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914684"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of NTCIR-7. 77--114","author":"Sakai Tetsuya","year":"2008","unstructured":"Tetsuya Sakai , Noriko Kando , Chuan-Jie Lin , Teruko Mitamura , Hideki Shima , Donghong Ji , Kuang-Hua Chen , and Eric Nyberg . 2008 . Overview of the NTCIR-7 ACLIA IR4QA Task . In Proceedings of NTCIR-7. 77--114 . Tetsuya Sakai, Noriko Kando, Chuan-Jie Lin, Teruko Mitamura, Hideki Shima, Donghong Ji, Kuang-Hua Chen, and Eric Nyberg. 2008. Overview of the NTCIR-7 ACLIA IR4QA Task. In Proceedings of NTCIR-7. 77--114."},{"key":"e_1_3_2_1_26_1","volume-title":"Overview of the NTCIR-12 Short Text Conversation Task Proceedings of NTCIR-12","author":"Shang Lifeng","year":"2016","unstructured":"Lifeng Shang , Tetsuya Sakai , Zhengdong Lu , Hang Li , Ryuichiro Higashinaka , and Yusuke Miyao 2016 . Overview of the NTCIR-12 Short Text Conversation Task Proceedings of NTCIR-12 . 473--484. Lifeng Shang, Tetsuya Sakai, Zhengdong Lu, Hang Li, Ryuichiro Higashinaka, and Yusuke Miyao 2016. Overview of the NTCIR-12 Short Text Conversation Task Proceedings of NTCIR-12. 473--484."},{"key":"e_1_3_2_1_27_1","volume-title":"A Comparison of Statistical Significance Tests for Information Retrieval Evaluation Proceedings of ACM CIKM","author":"Smucker Mark D.","year":"2007","unstructured":"Mark D. Smucker , James Allan , and Ben Carterette . 2007. A Comparison of Statistical Significance Tests for Information Retrieval Evaluation Proceedings of ACM CIKM 2007 . 623--632. Mark D. Smucker, James Allan, and Ben Carterette. 2007. A Comparison of Statistical Significance Tests for Information Retrieval Evaluation Proceedings of ACM CIKM 2007. 623--632."},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of NTCIR-9. 82--105","author":"Song Ruihua","year":"2011","unstructured":"Ruihua Song , Min Zhang , Tetsuya Sakai , Makoto P. Kato , Yiqun Liu , Miho Sugimoto , Qinglei Wang , and Naoki Orii . 2011 . Overview of the NTCIR-9 INTENT Task . In Proceedings of NTCIR-9. 82--105 . Ruihua Song, Min Zhang, Tetsuya Sakai, Makoto P. Kato, Yiqun Liu, Miho Sugimoto, Qinglei Wang, and Naoki Orii. 2011. Overview of the NTCIR-9 INTENT Task. In Proceedings of NTCIR-9. 82--105."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/6.1.1"},{"key":"e_1_3_2_1_30_1","unstructured":"Hideki Toyoda (Ed.). 2015. Fundamentals of Bayesian statistics: Practical Getting Started by Hamiltonian Monte Carlo Method (in Japanese). Asakura Shoten.  Hideki Toyoda (Ed.). 2015. Fundamentals of Bayesian statistics: Practical Getting Started by Hamiltonian Monte Carlo Method (in Japanese). Asakura Shoten."},{"key":"e_1_3_2_1_31_1","unstructured":"Hideki Toyoda. 2016. An Introduction to Statistical Data Analysis: Bayesian Statistics for 'post p-value era' (in Japanese). Asakuha Shoten.  Hideki Toyoda. 2016. An Introduction to Statistical Data Analysis: Bayesian Statistics for 'post p-value era' (in Japanese). Asakuha Shoten."},{"key":"e_1_3_2_1_32_1","volume-title":"Overview of the TREC 2003 Robust Retrieval Track Proceedings of TREC 2003.","author":"Voorhees Ellen M.","year":"2004","unstructured":"Ellen M. Voorhees . 2004 . Overview of the TREC 2003 Robust Retrieval Track Proceedings of TREC 2003. Ellen M. Voorhees. 2004. Overview of the TREC 2003 Robust Retrieval Track Proceedings of TREC 2003."},{"key":"e_1_3_2_1_33_1","volume-title":"Lazar","author":"Wasserstein Ronald L.","year":"2016","unstructured":"Ronald L. Wasserstein and Nicole A . Lazar 2016 . The ASA's Statement on P-values : Context, Process, and Purpose. The American Statistician ( 2016). Ronald L. Wasserstein and Nicole A. Lazar 2016. The ASA's Statement on P-values: Context, Process, and Purpose. The American Statistician (2016)."},{"key":"e_1_3_2_1_34_1","volume-title":"Bayesian Performance Comparison of Text Classifiers Proceedings of ACM SIGIR","author":"Zhang Dell","year":"2016","unstructured":"Dell Zhang , Jun Wang , Emine Yilmaz , Xiaoling Wang , and Yuxin Zhou 2016 . Bayesian Performance Comparison of Text Classifiers Proceedings of ACM SIGIR 2016. 15--24. Dell Zhang, Jun Wang, Emine Yilmaz, Xiaoling Wang, and Yuxin Zhou 2016. Bayesian Performance Comparison of Text Classifiers Proceedings of ACM SIGIR 2016. 15--24."},{"key":"e_1_3_2_1_35_1","volume-title":"McCloskey","author":"Ziliak Stephen T.","year":"2008","unstructured":"Stephen T. Ziliak and Deirdre N . McCloskey 2008 . The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. The University of Michigan Press . Stephen T. Ziliak and Deirdre N. McCloskey 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. The University of Michigan Press."}],"event":{"name":"SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Shinjuku Tokyo Japan","acronym":"SIGIR '17"},"container-title":["Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3077136.3080766","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3077136.3080766","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:04:07Z","timestamp":1750273447000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3077136.3080766"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,7]]},"references-count":35,"alternative-id":["10.1145\/3077136.3080766","10.1145\/3077136"],"URL":"https:\/\/doi.org\/10.1145\/3077136.3080766","relation":{},"subject":[],"published":{"date-parts":[[2017,8,7]]},"assertion":[{"value":"2017-08-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}