{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:21:20Z","timestamp":1750220480241,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":41,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003130","name":"Fonds Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["1S95020N"],"award-info":[{"award-number":["1S95020N"]}],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,10,18]]},"DOI":"10.1145\/3462244.3479889","type":"proceedings-article","created":{"date-parts":[[2021,10,15]],"date-time":"2021-10-15T14:41:47Z","timestamp":1634308907000},"page":"494-502","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures"],"prefix":"10.1145","author":[{"given":"Pieter","family":"Wolfert","sequence":"first","affiliation":[{"name":"Ghent University - imec, Belgium"}]},{"given":"Jeffrey M.","family":"Girard","sequence":"additional","affiliation":[{"name":"University of Kansas, USA"}]},{"given":"Taras","family":"Kucherenko","sequence":"additional","affiliation":[{"name":"KTH Royal Institute of Technology, Sweden"}]},{"given":"Tony","family":"Belpaeme","sequence":"additional","affiliation":[{"name":"Ghent University, Belgium"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.170"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Nichola Burton Michael Burton Dan Rigby Clare\u00a0AM Sutherland and Gillian Rhodes. 2019. Best-worst scaling improves measurement of first impressions. Cognitive research: principles and implications 4 1(2019) 1\u201310.  Nichola Burton Michael Burton Dan Rigby Clare\u00a0AM Sutherland and Gillian Rhodes. 2019. Best-worst scaling improves measurement of first impressions. Cognitive research: principles and implications 4 1(2019) 1\u201310.","DOI":"10.1186\/s41235-019-0183-2"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157689.2157798"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0190393"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0016956"},{"volume-title":"An Introduction to the Bootstrap","author":"Efron Bradley","key":"e_1_3_2_1_6_1","unstructured":"Bradley Efron and Robert\u00a0 J. Tibshirani . 1993. An Introduction to the Bootstrap . Chapman and Hall , New York, NY . Bradley Efron and Robert\u00a0J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman and Hall, New York, NY."},{"key":"e_1_3_2_1_7_1","volume-title":"Reliability of judgments of figural complexity.Journal of experimental psychology 56, 4","author":"Elliott Lois\u00a0Lawrence","year":"1958","unstructured":"Lois\u00a0Lawrence Elliott . 1958. Reliability of judgments of figural complexity.Journal of experimental psychology 56, 4 ( 1958 ), 335. Lois\u00a0Lawrence Elliott. 1958. Reliability of judgments of figural complexity.Journal of experimental psychology 56, 4 (1958), 335."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446802800108"},{"volume-title":"Linear mixed-effects model","author":"Ga\u0142ecki Andrzej","key":"e_1_3_2_1_9_1","unstructured":"Andrzej Ga\u0142ecki and Tomasz Burzykowski . 2013. Linear mixed-effects model . In Linear Mixed-Effects Models Using R. Springer , 245\u2013273. Andrzej Ga\u0142ecki and Tomasz Burzykowski. 2013. Linear mixed-effects model. In Linear Mixed-Effects Models Using R. Springer, 245\u2013273."},{"volume-title":"Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters","author":"Gwet Kilem\u00a0Li","key":"e_1_3_2_1_10_1","unstructured":"Kilem\u00a0Li Gwet . 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters ( fourth ed.). Advanced Analytics , Gaithersburg, MD . Kilem\u00a0Li Gwet. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters(fourth ed.). Advanced Analytics, Gaithersburg, MD."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-015-0280-4"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-011-9461-x"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383652.3423860"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3462244.3479957"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"#cr-split#-e_1_3_2_1_16_1.1","doi-asserted-by":"crossref","unstructured":"Taras Kucherenko Dai Hasegawa Naoshi Kaneko Gustav\u00a0Eje Henter and Hedvig Kjellstr\u00f6m. 2021. Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation. Int. J. Hum. Comput. Interact.(2021). https:\/\/doi.org\/10.1080\/10447318.2021.1883883 10.1080\/10447318.2021.1883883","DOI":"10.1080\/10447318.2021.1883883"},{"key":"#cr-split#-e_1_3_2_1_16_1.2","doi-asserted-by":"crossref","unstructured":"Taras Kucherenko Dai Hasegawa Naoshi Kaneko Gustav\u00a0Eje Henter and Hedvig Kjellstr\u00f6m. 2021. Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation. Int. J. Hum. Comput. Interact.(2021). https:\/\/doi.org\/10.1080\/10447318.2021.1883883","DOI":"10.1080\/10447318.2021.1883883"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382507.3418815"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450692"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v082.i13"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Weixin Liang James Zou and Zhou Yu. 2020. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. arXiv preprint arXiv:2005.10716(2020).  Weixin Liang James Zou and Zhou Yu. 2020. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. arXiv preprint arXiv:2005.10716(2020).","DOI":"10.18653\/v1\/2020.acl-main.126"},{"key":"e_1_3_2_1_21_1","volume-title":"Communicating to learn: Infants","author":"Lucca Kelsey","year":"2018","unstructured":"Kelsey Lucca and Makeba\u00a0Parramore Wilbourn . 2018. Communicating to learn: Infants \u2019 pointing gestures result in optimal learning. Child de velopment 89, 3 ( 2018 ), 941\u2013960. Kelsey Lucca and Makeba\u00a0Parramore Wilbourn. 2018. Communicating to learn: Infants\u2019 pointing gestures result in optimal learning. Child development 89, 3 (2018), 941\u2013960."},{"key":"e_1_3_2_1_22_1","volume-title":"Don\u2019t Classify Ratings of Affect","author":"Martinez Hector","year":"2014","unstructured":"Hector Martinez , Georgios Yannakakis , and John Hallam . 2014. Don\u2019t Classify Ratings of Affect ; Rank Them!IEEE Transactions on Affective Computing 3045, c ( 2014 ), 1\u20131. https:\/\/doi.org\/10\/f6pnzt Hector Martinez, Georgios Yannakakis, and John Hallam. 2014. Don\u2019t Classify Ratings of Affect; Rank Them!IEEE Transactions on Affective Computing 3045, c (2014), 1\u20131. https:\/\/doi.org\/10\/f6pnzt"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.1.1.30"},{"volume-title":"Hand and mind: What gestures reveal about thought","author":"McNeill David","key":"e_1_3_2_1_24_1","unstructured":"David McNeill . 1992. Hand and mind: What gestures reveal about thought . University of Chicago press. David McNeill. 1992. Hand and mind: What gestures reveal about thought. University of Chicago press."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Kim\u00a0T Mueser Barry\u00a0W Grau Steve Sussman and Alexander\u00a0J Rosen. 1984. You\u2019re only as pretty as you feel: facial expression as a determinant of physical attractiveness.Journal of Personality and Social Psychology 46 2(1984) 469.  Kim\u00a0T Mueser Barry\u00a0W Grau Steve Sussman and Alexander\u00a0J Rosen. 1984. You\u2019re only as pretty as you feel: facial expression as a determinant of physical attractiveness.Journal of Personality and Social Psychology 46 2(1984) 469.","DOI":"10.1037\/0022-3514.46.2.469"},{"key":"e_1_3_2_1_26_1","unstructured":"Laura P\u00e9rez-Mayos Mireia Farr\u00fas and Jordi Adell. 2019. Part-of-speech and prosody-based approaches for robot speech and gesture synchronization. Journal of Intelligent & Robotic Systems(2019) 1\u201311.  Laura P\u00e9rez-Mayos Mireia Farr\u00fas and Jordi Adell. 2019. Part-of-speech and prosody-based approaches for robot speech and gesture synchronization. Journal of Intelligent & Robotic Systems(2019) 1\u201311."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.14.13022"},{"key":"e_1_3_2_1_28_1","volume-title":"Beat gestures improve word recall in 3-to 5-year-old children. Journal of Experimental Child Psychology. 2017 Apr","author":"Prieto\u00a0Vives Pilar","year":"2017","unstructured":"Pilar Prieto\u00a0Vives , Alfonso Igualada\u00a0P\u00e9rez , and N\u00faria Esteve\u00a0Gibert . 2017. Beat gestures improve word recall in 3-to 5-year-old children. Journal of Experimental Child Psychology. 2017 Apr ; 156: 99-112 ( 2017 ). Pilar Prieto\u00a0Vives, Alfonso Igualada\u00a0P\u00e9rez, and N\u00faria Esteve\u00a0Gibert. 2017. Beat gestures improve word recall in 3-to 5-year-old children. Journal of Experimental Child Psychology. 2017 Apr; 156: 99-112 (2017)."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0029315"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-013-0196-9"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5334\/jors.187"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1037\/pas0000648"},{"key":"e_1_3_2_1_33_1","volume-title":"The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): a new technique for psychological measurement. Behavior research methods 50, 4","author":"Sung Yao-Ting","year":"2018","unstructured":"Yao-Ting Sung and Jeng-Shin Wu. 2018. The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): a new technique for psychological measurement. Behavior research methods 50, 4 ( 2018 ), 1694\u20131715. Yao-Ting Sung and Jeng-Shin Wu. 2018. The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): a new technique for psychological measurement. Behavior research methods 50, 4 (2018), 1694\u20131715."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.21437\/SSW.2019-19"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2010.02.004"},{"key":"e_1_3_2_1_36_1","volume-title":"ICDL-EPIROB 2019 Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions.","author":"Wolfert Pieter","year":"2019","unstructured":"Pieter Wolfert , Taras Kucherenko , Hedvig Kjellstr\u00f6m , and Tony Belpaeme . 2019 . Should Beat Gestures Be Learned Or Designed?: A Benchmarking User Study . In ICDL-EPIROB 2019 Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions. Pieter Wolfert, Taras Kucherenko, Hedvig Kjellstr\u00f6m, and Tony Belpaeme. 2019. Should Beat Gestures Be Learned Or Designed?: A Benchmarking User Study. In ICDL-EPIROB 2019 Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions."},{"key":"e_1_3_2_1_37_1","unstructured":"Pieter Wolfert Nicole Robinson and Tony Belpaeme. 2021. A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents. arXiv preprint arXiv:2101.03769(2021).  Pieter Wolfert Nicole Robinson and Tony Belpaeme. 2021. A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents. arXiv preprint arXiv:2101.03769(2021)."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2018.2879512"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2015.7344627"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417838"}],"event":{"name":"ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"Montr\u00e9al QC Canada","acronym":"ICMI '21"},"container-title":["Proceedings of the 2021 International Conference on Multimodal Interaction"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3462244.3479889","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3462244.3479889","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:54Z","timestamp":1750193334000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3462244.3479889"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,18]]},"references-count":41,"alternative-id":["10.1145\/3462244.3479889","10.1145\/3462244"],"URL":"https:\/\/doi.org\/10.1145\/3462244.3479889","relation":{},"subject":[],"published":{"date-parts":[[2021,10,18]]},"assertion":[{"value":"2021-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}