{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T19:52:31Z","timestamp":1782417151617,"version":"3.54.5"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2020,5,28]],"date-time":"2020-05-28T00:00:00Z","timestamp":1590624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,5,28]]},"abstract":"<jats:p>Crowdsourcing marketplaces have provided a large number of opportunities for online workers to earn a living. To improve satisfaction and engagement of such workers, who are vital for the sustainability of the marketplaces, recent works have used conversational interfaces to support the execution of a variety of crowdsourcing tasks. The rationale behind using conversational interfaces stems from the potential engagement that conversation can stimulate. Prior works in psychology have also shown that conversational styles can play an important role in communication. There are unexplored opportunities to estimate a worker's conversational style with an end goal of improving worker satisfaction, engagement and quality. Addressing this knowledge gap, we investigate the role of conversational styles in conversational microtask crowdsourcing. To this end, we design a conversational interface which supports task execution, and we propose methods to estimate the conversational style of a worker. Our experimental setup was designed to empirically observe how conversational styles of workers relate with quality-related outcomes. Results show that even a naive supervised classifier can predict the conversation style with high accuracy (80%), and crowd workers with an Involvement conversational style provided a significantly higher output quality, exhibited a higher user engagement and perceived less cognitive task load in comparison to their counterparts. Our findings have important implications on task design with respect to improving worker performance and their engagement in microtask crowdsourcing.<\/jats:p>","DOI":"10.1145\/3392837","type":"journal-article","created":{"date-parts":[[2020,5,29]],"date-time":"2020-05-29T16:01:06Z","timestamp":1590768066000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Estimating Conversational Styles in Conversational Microtask Crowdsourcing"],"prefix":"10.1145","volume":"4","author":[{"given":"Sihang","family":"Qiu","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Bozzon","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,5,29]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1361684.1361685"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3176349.3176380"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13740-013-0032-2"},{"key":"e_1_2_2_4_1","first-page":"33","article-title":"Curious Cat--Mobile, Context-Aware Conversational Crowdsourcing Knowledge Acquisition","volume":"35","author":"Luka Bradevs","year":"2017","unstructured":"Luka Bradevs ko, Michael Witbrock , Janez Starc , Zala Herga , Marko Grobelnik , and Dunja Mladeni\u0107 . 2017 . Curious Cat--Mobile, Context-Aware Conversational Crowdsourcing Knowledge Acquisition . ACM Transactions on Information Systems (TOIS) , Vol. 35 , 4 (2017), 33 . Luka Bradevs ko, Michael Witbrock, Janez Starc, Zala Herga, Marko Grobelnik, and Dunja Mladeni\u0107. 2017. Curious Cat--Mobile, Context-Aware Conversational Crowdsourcing Knowledge Acquisition. ACM Transactions on Information Systems (TOIS), Vol. 35, 4 (2017), 33.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702329"},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 1301--1309","author":"Burger John D","year":"2011","unstructured":"John D Burger , John Henderson , George Kim , and Guido Zarrella . 2011 . Discriminating gender on Twitter . In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 1301--1309 . John D Burger, John Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on Twitter. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 1301--1309."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1086\/671465"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2886425"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3098279.3098539"},{"key":"e_1_2_2_10_1","volume-title":"Ujwal Gadiraju, Michele Catasta, et al.","author":"Demartini Gianluca","year":"2017","unstructured":"Gianluca Demartini , Djellel Eddine Difallah , Ujwal Gadiraju, Michele Catasta, et al. 2017 . An introduction to hybrid human-machine information systems. Foundations and Trends\u00ae in Web Science , Vol. 7 , 1 (2017), 1--87. Gianluca Demartini, Djellel Eddine Difallah, Ujwal Gadiraju, Michele Catasta, et al. 2017. An introduction to hybrid human-machine information systems. Foundations and Trends\u00ae in Web Science, Vol. 7, 1 (2017), 1--87."},{"key":"e_1_2_2_11_1","volume-title":"Increasing cheat robustness of crowdsourcing tasks. Information retrieval","author":"Eickhoff Carsten","year":"2013","unstructured":"Carsten Eickhoff and Arjen P de Vries . 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval , Vol. 16 , 2 ( 2013 ), 121--137. Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, Vol. 16, 2 (2013), 121--137."},{"key":"e_1_2_2_12_1","volume-title":"Measuring nominal scale agreement among many raters. Psychological bulletin","author":"Fleiss Joseph L","year":"1971","unstructured":"Joseph L Fleiss . 1971. Measuring nominal scale agreement among many raters. Psychological bulletin , Vol. 76 , 5 ( 1971 ), 378. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, Vol. 76, 5 (1971), 378."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027402"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093650209351468"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2836041.2841208"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300249"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291035"},{"key":"e_1_2_2_20_1","volume-title":"2019 b. The impact of task abandonment in crowdsourcing","author":"Han Lei","year":"2019","unstructured":"Lei Han , Kevin Roitero , Ujwal Gadiraju , Cristina Sarasua , Alessandro Checco , Eddy Maddalena , and Gianluca Demartini . 2019 b. The impact of task abandonment in crowdsourcing . IEEE Transactions on Knowledge and Data Engineering ( 2019 ). Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019 b. The impact of task abandonment in crowdsourcing. IEEE Transactions on Knowledge and Data Engineering (2019)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173869"},{"key":"e_1_2_2_22_1","volume-title":"Third AAAI conference on human computation and crowdsourcing.","author":"Kenneth Huang Ting-Hao","year":"2015","unstructured":"Ting-Hao Kenneth Huang , Walter S Lasecki , and Jeffrey P Bigham . 2015 . Guardian: A crowd-powered spoken dialog system for web apis . In Third AAAI conference on human computation and crowdsourcing. Ting-Hao Kenneth Huang, Walter S Lasecki, and Jeffrey P Bigham. 2015. Guardian: A crowd-powered spoken dialog system for web apis. In Third AAAI conference on human computation and crowdsourcing."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_2_2_24_1","first-page":"79","article-title":"Conversations-and negotiated interaction-in text and voice chat rooms","volume":"9","author":"Jepson Kevin","year":"2005","unstructured":"Kevin Jepson . 2005 . Conversations-and negotiated interaction-in text and voice chat rooms . Language Learning & Technology , Vol. 9 , 3 (2005), 79 -- 98 . Kevin Jepson. 2005. Conversations-and negotiated interaction-in text and voice chat rooms. Language Learning & Technology, Vol. 9, 3 (2005), 79--98.","journal-title":"Language Learning & Technology"},{"key":"e_1_2_2_25_1","volume-title":"Proc. Alexa Prize","author":"Jonell Patrik","year":"2018","unstructured":"Patrik Jonell , Mattias Bystedt , Fethiye Irmak Dogan , Per Fallgren , Jonas Ivarsson , Marketa Slukova , Jos\u00e9 Lopes Ulme Wennberg , Johan Boye , and Gabriel Skantze . 2018 . Fantom: A Crowdsourced Social Chatbot using an Evolving Dialog Graph . Proc. Alexa Prize (2018). Patrik Jonell, Mattias Bystedt, Fethiye Irmak Dogan, Per Fallgren, Jonas Ivarsson, Marketa Slukova, Jos\u00e9 Lopes Ulme Wennberg, Johan Boye, and Gabriel Skantze. 2018. Fantom: A Crowdsourced Social Chatbot using an Evolving Dialog Graph. Proc. Alexa Prize (2018)."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998196"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300316"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2854961"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741681"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2018.252095348"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1749-6632.1979.tb17753.x"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502057"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_2_2_36_1","volume-title":"First AAAI Conference on Human Computation and Crowdsourcing.","author":"Mao Andrew","year":"2013","unstructured":"Andrew Mao , Ece Kamar , and Eric Horvitz . 2013 . Why stop now? predicting worker engagement in online crowdsourcing . In First AAAI Conference on Human Computation and Crowdsourcing. Andrew Mao, Ece Kamar, and Eric Horvitz. 2013. Why stop now? predicting worker engagement in online crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3320435.3320439"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3027077"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531628"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2107636.2107651"},{"key":"e_1_2_2_41_1","volume-title":"Why Engagement Matters","author":"O'Brien Heather","unstructured":"Heather O'Brien . 2016. Theoretical perspectives on user engagement . In Why Engagement Matters . Springer , 1--26. Heather O'Brien. 2016. Theoretical perspectives on user engagement. In Why Engagement Matters. Springer, 1--26."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2018.01.004"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1517744.1517760"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376403"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1076\/0958-8221(200002)13:1;1-K;FT005"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819985"},{"key":"e_1_2_2_47_1","unstructured":"Jonathan Schler Moshe Koppel Shlomo Argamon and James W Pennebaker. 2006. Effects of age and gender on blogging.. In AAAI spring symposium: Computational approaches to analyzing weblogs. 199--205.  Jonathan Schler Moshe Koppel Shlomo Argamon and James W Pennebaker. 2006. Effects of age and gender on blogging.. In AAAI spring symposium: Computational approaches to analyzing weblogs. 199--205."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47665-0_4"},{"key":"e_1_2_2_49_1","volume-title":"Conversational style. Psycholinguistic models of production","author":"Tannen Deborah","year":"1987","unstructured":"Deborah Tannen . 1987. Conversational style. Psycholinguistic models of production ( 1987 ), 251--267. Deborah Tannen. 1987. Conversational style. Psycholinguistic models of production (1987), 251--267."},{"key":"e_1_2_2_50_1","volume-title":"Conversational style: Analyzing talk among friends","author":"Tannen Deborah","unstructured":"Deborah Tannen . 2005. Conversational style: Analyzing talk among friends . Oxford University Press . Deborah Tannen. 2005. Conversational style: Analyzing talk among friends .Oxford University Press."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470720"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3176349.3176388"},{"key":"e_1_2_2_53_1","volume-title":"SIGIR 1st International Workshop on Conversational Approaches to Information Retrieval (CAIR'17)","volume":"5","author":"Thomas Paul","year":"2017","unstructured":"Paul Thomas , Daniel McDuff , Mary Czerwinski , and Nick Craswell . 2017 . MISC: A data set of information-seeking conversations . In SIGIR 1st International Workshop on Conversational Approaches to Information Retrieval (CAIR'17) , Vol. 5 . Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2017. MISC: A data set of information-seeking conversations. In SIGIR 1st International Workshop on Conversational Approaches to Information Retrieval (CAIR'17), Vol. 5."},{"key":"e_1_2_2_54_1","volume-title":"Collective Intelligence","author":"Toxtli Carlos","year":"2018","unstructured":"Carlos Toxtli , Joel Chan , Walter S Lasecki , and Saiph Savage . 2018. Enabling Expert Critique with Chatbots and Micro Guidance . In Collective Intelligence 2018 . ACM , 4. Carlos Toxtli, Joel Chan, Walter S Lasecki, and Saiph Savage. 2018. Enabling Expert Critique with Chatbots and Micro Guidance. In Collective Intelligence 2018. ACM, 4."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3052767"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053175"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13283"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300772"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSCWD.2018.8465305"},{"key":"e_1_2_2_61_1","first-page":"26","article-title":"Dissonance Between Human and Machine Understanding","volume":"56","author":"Zhang Zijian","year":"2019","unstructured":"Zijian Zhang , Jaspreet Singh , Ujwal Gadiraju , and Avishek Anand . 2019 . Dissonance Between Human and Machine Understanding . Proc. ACM Hum.-Comput. Interact. 3 CSCW , Vol. 56 (2019), 26 . Zijian Zhang, Jaspreet Singh, Ujwal Gadiraju, and Avishek Anand. 2019. Dissonance Between Human and Machine Understanding. Proc. ACM Hum.-Comput. Interact. 3 CSCW, Vol. 56 (2019), 26.","journal-title":"Proc. ACM Hum.-Comput. Interact. 3 CSCW"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326010"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392837","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3392837","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:16Z","timestamp":1750200076000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,28]]},"references-count":62,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2020,5,28]]}},"alternative-id":["10.1145\/3392837"],"URL":"https:\/\/doi.org\/10.1145\/3392837","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,28]]},"assertion":[{"value":"2020-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}