{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T22:55:47Z","timestamp":1771368947411,"version":"3.50.1"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,6,28]],"date-time":"2018-06-28T00:00:00Z","timestamp":1530144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2018,6,30]]},"abstract":"<jats:p>Discovering gestures that gain consensus is a key goal of gesture elicitation. To this end, HCI research has developed statistical methods to reason about agreement. We review these methods and identify three major problems. First, we show that raw agreement rates disregard agreement that occurs by chance and do not reliably capture how participants distinguish among referents. Second, we explain why current recommendations on how to interpret agreement scores rely on problematic assumptions. Third, we demonstrate that significance tests for comparing agreement rates, either within or between participants, yield large Type I error rates (&gt;40% for \u03b1 =.05). As alternatives, we present agreement indices that are routinely used in inter-rater reliability studies. We discuss how to apply them to gesture elicitation studies. We also demonstrate how to use common resampling techniques to support statistical inference with interval estimates. We apply these methods to reanalyze and reinterpret the findings of four gesture elicitation studies.<\/jats:p>","DOI":"10.1145\/3182168","type":"journal-article","created":{"date-parts":[[2018,6,28]],"date-time":"2018-06-28T16:37:19Z","timestamp":1530203839000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":51,"title":["Fallacies of Agreement"],"prefix":"10.1145","volume":"25","author":[{"given":"Theophanis","family":"Tsandilas","sequence":"first","affiliation":[{"name":"Inria, Universit\u00e9 Paris-Saclay, and Univ. Paris-Sud, Orsay Cedex, France"}]}],"member":"320","published-online":{"date-parts":[[2018,6,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-230-36355-7"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470734"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858159"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316448104100307"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858589"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.1180"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90159-M"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/37.3-4.256"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240723"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2011.10.017"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.biopsych.2004.10.016"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","volume-title":"Fair Statistical Communication in HCI","author":"Dragicevic Pierre","DOI":"10.1007\/978-3-319-26633-6_13"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344552"},{"key":"e_1_2_1_17_1","unstructured":"David Ellerman. 2010. History of the Logical Entropy Formula. Retrieved from http:\/\/www.ellerman.org\/history-of-the-logical-entropy-formula\/.  David Ellerman. 2010. History of the Logical Entropy Formula. Retrieved from http:\/\/www.ellerman.org\/history-of-the-logical-entropy-formula\/."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90158-L"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208660"},{"key":"e_1_2_1_20_1","volume-title":"Statistical Methods for Research Workers","author":"Fisher Ronald Aylmer","edition":"20"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_2_1_22_1","volume-title":"Origins of Sound Change: Approaches to Phonologization. Alan C. L. Yu (Ed.)","author":"Garrett Andrew"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/2447556.2447679"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/358274.358284"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2669485.2669511"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11336-007-9054-8"},{"key":"e_1_2_1_27_1","volume-title":"Handbook of Inter-Rater Reliability","author":"Gwet Kilem Li","edition":"4"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1530064.1530066"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312450709336664"},{"key":"e_1_2_1_30_1","volume-title":"Introduction to the Practice of Statistics","author":"Hesterberg Tim"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557004"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v028.i08"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208557"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2886442"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2729103"},{"key":"e_1_2_1_37_1","first-page":"411","article-title":"Reliability in content analysis: Some common misconceptions and recommendations","volume":"30","author":"Krippendorff Klaus","year":"2004","journal-title":"Human Communication Research"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312458.2011.568376"},{"key":"e_1_2_1_39_1","volume-title":"Content Analysis: An Introduction to its Methodology","author":"Krippendorff Klaus","year":"2013"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979136"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753572"},{"key":"e_1_2_1_42_1","first-page":"31","article-title":"Codebook development for team-based qualitative analysis","volume":"10","author":"MacQueen Kathleen M.","year":"1998","journal-title":"Cultural Anthropology Methods"},{"key":"e_1_2_1_43_1","volume-title":"Information Theory and Psycholinguistics: A Theory of Word Frequencies","author":"Mandelbrot Benoit"},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","volume-title":"The Whole-object, Taxonomic, and Mutual Exclusivity Assumptions as Initial Constraints on Word Meanings","author":"Markman Ellen M.","DOI":"10.1017\/CBO9780511983689.004"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1731903.1731912"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396636.2396651"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2591689"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of Graphics Interface 2010 (GI\u201910)","author":"Morris Meredith Ringel"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1080\/00107510500052444"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","volume-title":"A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI","author":"Nielsen Michael","DOI":"10.1007\/978-3-540-24598-8_38"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.2307\/2531148"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466145"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1475-6773.2005.00444.x"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-014-0585-6"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468527"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.4780090917"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729989"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753458"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702583"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1086\/266577"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1038\/163688a0"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.125.4.341"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2598153.2598184"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/0022-3956(82)90039-5"},{"key":"e_1_2_1_66_1","unstructured":"John S. Uebersax. 2015. Statistical Methods for Diagnostic Agreement. Retrieved from http:\/\/www.john-uebersax.com\/stat\/agree.htm.  John S. Uebersax. 2015. Statistical Methods for Diagnostic Agreement. Retrieved from http:\/\/www.john-uebersax.com\/stat\/agree.htm."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11336-009-9116-1"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702223"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858228"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208391"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557239"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2212776.2212419"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/1056808.1057043"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518866"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094428105280059"},{"key":"e_1_2_1_76_1","volume-title":"Human Behaviour and the Principle of Least Effort","author":"Zipf George K."}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3182168","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3182168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:41:19Z","timestamp":1750282879000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3182168"}},"subtitle":["A Critical Review of Consensus Assessment Methods for Gesture Elicitation"],"short-title":[],"issued":{"date-parts":[[2018,6,28]]},"references-count":74,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,6,30]]}},"alternative-id":["10.1145\/3182168"],"URL":"https:\/\/doi.org\/10.1145\/3182168","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,28]]},"assertion":[{"value":"2017-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}