{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,31]],"date-time":"2025-07-31T00:41:18Z","timestamp":1753922478863,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":33,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T00:00:00Z","timestamp":1603065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP)"},{"name":"Stiftelsen f\u00f6r Strategisk Forskning","award":["RIT15-0107"],"award-info":[{"award-number":["RIT15-0107"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,20]]},"DOI":"10.1145\/3383652.3423860","type":"proceedings-article","created":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T22:10:31Z","timestamp":1603145431000},"page":"1-8","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Can we trust online crowdworkers?"],"prefix":"10.1145","author":[{"given":"Patrik","family":"Jonell","sequence":"first","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taras","family":"Kucherenko","sequence":"additional","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ilaria","family":"Torre","sequence":"additional","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonas","family":"Beskow","sequence":"additional","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,10,19]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Can we turn shirkers into workers? Journal of Experimental Social Psychology 66","author":"Berinsky Adam J","year":"2016","unstructured":"Adam J Berinsky , Michele F Margolis , and Michael W Sances . 2016. Can we turn shirkers into workers? Journal of Experimental Social Psychology 66 ( 2016 ). Adam J Berinsky, Michele F Margolis, and Michael W Sances. 2016. Can we turn shirkers into workers? Journal of Experimental Social Psychology 66 (2016)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2015.2493525"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcomdis.2014.11.003"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-016-9458-5"},{"volume-title":"Statistical power analysis for the behavioral sciences","author":"Cohen Jacob","key":"e_1_3_2_1_5_1","unstructured":"Jacob Cohen . 2013. Statistical power analysis for the behavioral sciences . Academic press . Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic press."},{"volume-title":"The Revised NEO Personality Inventory (NEO-PI-R)","author":"Costa Paul T","key":"e_1_3_2_1_6_1","unstructured":"Paul T Costa Jr and Robert R McCrae . 2008. The Revised NEO Personality Inventory (NEO-PI-R) . Sage Publications, Inc. Paul T Costa Jr and Robert R McCrae. 2008. The Revised NEO Personality Inventory (NEO-PI-R). Sage Publications, Inc."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0057410"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2015.07.006"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-017-9514-9"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1017\/iop.2015.25"},{"key":"e_1_3_2_1_11_1","volume-title":"Is the Web as good as the lab? Comparable performance from Web and lab in cognitive\/perceptual experiments. Psychonomic bulletin & review 19, 5","author":"Germine Laura","year":"2012","unstructured":"Laura Germine , Ken Nakayama , Bradley C Duchaine , Christopher F Chabris , Garga Chatterjee , and Jeremy B Wilmer . 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive\/perceptual experiments. Psychonomic bulletin & review 19, 5 ( 2012 ), 847--857. Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer. 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive\/perceptual experiments. Psychonomic bulletin & review 19, 5 (2012), 847--857."},{"key":"e_1_3_2_1_12_1","unstructured":"Jeffrey Girard. 2020. agreement. R package version 0.0.0.9002.  Jeffrey Girard. 2020. agreement. R package version 0.0.0.9002."},{"key":"e_1_3_2_1_13_1","unstructured":"Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics LLC.  Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics LLC."},{"key":"e_1_3_2_1_14_1","volume-title":"Are manipulation checks necessary? Frontiers in psychology 9","author":"Hauser David J","year":"2018","unstructured":"David J Hauser , Phoebe C Ellsworth , and Richard Gonzalez . 2018. Are manipulation checks necessary? Frontiers in psychology 9 ( 2018 ). David J Hauser, Phoebe C Ellsworth, and Richard Gonzalez. 2018. Are manipulation checks necessary? Frontiers in psychology 9 (2018)."},{"key":"e_1_3_2_1_15_1","volume-title":"Attentive Turkers: M Turk participants perform better on online attention checks than do subject pool participants. Behavior research methods 48, 1","author":"Hauser David J","year":"2016","unstructured":"David J Hauser and Norbert Schwarz . 2016 . Attentive Turkers: M Turk participants perform better on online attention checks than do subject pool participants. Behavior research methods 48, 1 (2016), 400--407. David J Hauser and Norbert Schwarz. 2016. Attentive Turkers: M Turk participants perform better on online attention checks than do subject pool participants. Behavior research methods 48, 1 (2016), 400--407."},{"key":"e_1_3_2_1_16_1","volume-title":"ICDL-EpiRob Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions","author":"Jonell Patrik","year":"2019","unstructured":"Patrik Jonell , Taras Kucherenko , Erik Ekstedt , and Jonas Beskow . 2019 . Learning Non-verbal Behavior for a Social Robot from YouTube Videos . In ICDL-EpiRob Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions , Oslo, Norway , August 19, 2019. Patrik Jonell, Taras Kucherenko, Erik Ekstedt, and Jonas Beskow. 2019. Learning Non-verbal Behavior for a Social Robot from YouTube Videos. In ICDL-EpiRob Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions, Oslo, Norway, August 19, 2019."},{"key":"e_1_3_2_1_17_1","volume-title":"AMCIS","volume":"11","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker Motivation in Crowd so urcing-A Study on Mechanical Turk .. In AMCIS , Vol. 11 . Detroit, Michigan, USA, 1--11. Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowd so urcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. Detroit, Michigan, USA, 1--11."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/00913367.2016.1269304"},{"volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Komarov Steven","key":"e_1_3_2_1_19_1","unstructured":"Steven Komarov , Katharina Reinecke , and Krzysztof Z. Gajos . 2013. Crowdsourcing Performance Evaluations of User Interfaces . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ( Paris, France) (CHI '13). ACM, New York, NY, USA, 207--216. https:\/\/doi.org\/10.1145\/2470654.2470684 10.1145\/2470654.2470684 Steven Komarov, Katharina Reinecke, and Krzysztof Z. Gajos. 2013. Crowdsourcing Performance Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). ACM, New York, NY, USA, 207--216. https:\/\/doi.org\/10.1145\/2470654.2470684"},{"key":"e_1_3_2_1_20_1","volume-title":"Analyzing Input and Output Representations for Speech-Driven Gesture Generation. In International Conference on Intelligent Virtual Agents (IVA '19)","volume":"19","author":"Kucherenko Taras","year":"2019","unstructured":"Taras Kucherenko , Dai Hasegawa , Gustav Eje Henter , Naoshi Kaneko , and Hedvig Kjellstr\u00f6m . 2019 . Analyzing Input and Output Representations for Speech-Driven Gesture Generation. In International Conference on Intelligent Virtual Agents (IVA '19) , Vol. 19 . ACM, Paris, France, 97--104. https:\/\/doi.org\/10.1145\/3308532.3329472 10.1145\/3308532.3329472 Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, and Hedvig Kjellstr\u00f6m. 2019. Analyzing Input and Output Representations for Speech-Driven Gesture Generation. In International Conference on Intelligent Virtual Agents (IVA '19), Vol. 19. ACM, Paris, France, 97--104. https:\/\/doi.org\/10.1145\/3308532.3329472"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382507.3418815"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/apps.12108"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v082.i13"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1044\/2015_AJSLP-15-0059"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/apps.12124"},{"key":"e_1_3_2_1_26_1","volume-title":"Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality","author":"Maniaci Michael R","year":"2014","unstructured":"Michael R Maniaci and Ronald D Rogge . 2014. Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality ( 2014 ). Michael R Maniaci and Ronald D Rogge. 2014. Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality (2014)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858321"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX.2015.7148091"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2009.03.009"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2019.04.005"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Margarita Stolarova Corinna Wolf Tanja Rinker and Aenne Brielmann. 2014. How to assess and compare inter-rater reliability agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs. Frontiers in psychology 5 (2014) 509.  Margarita Stolarova Corinna Wolf Tanja Rinker and Aenne Brielmann. 2014. How to assess and compare inter-rater reliability agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs. Frontiers in psychology 5 (2014) 509.","DOI":"10.3389\/fpsyg.2014.00509"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.1058"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793720"}],"event":{"name":"IVA '20: ACM International Conference on Intelligent Virtual Agents","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence"],"location":"Virtual Event Scotland UK","acronym":"IVA '20"},"container-title":["Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383652.3423860","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383652.3423860","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:56Z","timestamp":1750195916000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383652.3423860"}},"subtitle":["Comparing online and offline participants in a preference test of virtual agents"],"short-title":[],"issued":{"date-parts":[[2020,10,19]]},"references-count":33,"alternative-id":["10.1145\/3383652.3423860","10.1145\/3383652"],"URL":"https:\/\/doi.org\/10.1145\/3383652.3423860","relation":{},"subject":[],"published":{"date-parts":[[2020,10,19]]},"assertion":[{"value":"2020-10-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}